Commit 7e79193
committed
feat: Enhance parallel execution; resume optimization & gap detection
- Add resume optimization that adjusts min_block based on persistent
state
- Implement gap-aware partitioning for intelligent backfill
- Add pre-flight table creation to avoid locking issues
- Improve error handling and logging for state operations
- Support label joining in parallel workers
Key features:
- Auto-detects processed ranges and skips already-loaded partitions
- Prioritizes gap filling before processing new data
- Efficient partition creation avoiding redundant work
- Visible logging for resume operations and adjustments
Resume workflow:
1. Query state store for max processed block
2. Adjust min_block to skip processed ranges
3. Detect gaps in processed data
4. Create partitions prioritizing gaps first
5. Process remaining historical data1 parent 568423f commit 7e79193
1 file changed
+404
-34
lines changed
0 commit comments