Skip to content

Commit b61cc79

Browse files
committed
streaming: Enhance parallel execution; resumability & gap detection
- Add resume optimization that adjusts min_block based on persistent state - Implement gap-aware partitioning for intelligent backfill - Add pre-flight table creation to avoid locking issues - Improve error handling and logging for state operations - Support label joining in parallel workers Key features: - Auto-detects processed ranges and skips already-loaded partitions - Prioritizes gap filling before processing new data - Efficient partition creation avoiding redundant work - Visible logging for resume operations and adjustments Resume workflow: 1. Query state store for max processed block 2. Adjust min_block to skip processed ranges 3. Detect gaps in processed data 4. Create partitions prioritizing gaps first 5. Process remaining historical data
1 parent 70689cb commit b61cc79

File tree

1 file changed

+404
-34
lines changed

1 file changed

+404
-34
lines changed

0 commit comments

Comments
 (0)