Skip to content

Conversation

@ankitm111
Copy link

@ankitm111 ankitm111 commented Jan 27, 2026

Refactor tempest-hybrid to client wrapper architecture with enhanced S3 integration

Major improvements:

  • Replace complex proxy hierarchy with clean S3AwareDynamoDbClient wrapper
  • Add parallel S3 fetching for significant performance gains (10x for multi-item queries)
  • Implement configurable retry logic with exponential backoff for S3 operations
  • Add keyPrefix support for flexible S3 object organization
  • Introduce comprehensive metrics tracking (hydration latency, success/failure rates)
  • Improve error handling with dedicated S3HydrationException

Architecture changes:

  • Remove proxy classes (HybridDbProxy, HybridTableProxy, HybridViewProxy)
  • Add HybridDynamoDbFactory for simplified client creation
  • Centralize S3 operations in S3AwareDynamoDbClient
  • Support both managed and external executor lifecycles

Documentation:

  • Add MIGRATION.md guide for upgrading from proxy approach
  • Create comprehensive USAGE_EXAMPLES.md with real-world scenarios
  • Update README with clear quick-start guide

Testing:

  • Add test coverage for batch operations, error handling, metrics
  • Implement retry logic and executor lifecycle tests
  • Create integration tests for S3-aware client behavior

This refactoring simplifies the library's usage while providing better performance,
observability, and reliability for DynamoDB + S3 hybrid storage patterns.

ankitm111 and others added 8 commits October 31, 2025 15:02
This commit enhances the tempest-hybrid library with two key improvements:

1. **Functional keyPrefix Configuration**
   - Added new overloaded S3KeyGenerator.generateS3Key() method that accepts HybridConfig
   - keyPrefix from config is now prepended to generated S3 keys during archival
   - Maintains backward compatibility with existing method signature
   - Added comprehensive tests for prefix handling

2. **Optional Retry Logic for S3 Operations**
   - Added RetryConfig with configurable retry behavior
   - Implemented exponential backoff with jitter to prevent thundering herd
   - Smart exception handling - only retries transient network errors
   - Added RetryAttempt metric event for monitoring retry behavior
   - Disabled by default for backward compatibility
   - Added comprehensive test suite (7 tests) for retry scenarios

Both features are fully backward compatible and include extensive documentation updates.

Tests: All 60 tempest-hybrid tests passing
Docs: Updated README, MIGRATION, and USAGE_EXAMPLES with new features

Co-Authored-By: Claude Opus 4.1 <[email protected]>
Copy link
Collaborator

@kyeotic kyeotic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review is to block merging until a more complete review can be done.

My initial impression is that this PR is not a good fit for tempest. It adds a new module with no apparent API overlap from tempest. It does not appear to follow tempest patterns either.

@kyeotic kyeotic self-assigned this Jan 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants