Commit 568423f
committed
refactor: Major base loader improvements for streaming and resilience
- Integrate state management for resume and deduplication
- Add label joining support with automatic type conversion
- Implement resilience features (retry, backpressure, rate limiting)
- Add metadata columns (_amp_batch_id) for reorg handling
- Support streaming with block ranges and reorg detection
- Separate _try_load_batch() for better error handling1 parent 7ab882b commit 568423f
File tree
5 files changed
+695
-95
lines changed- src/amp
- loaders
- streaming
5 files changed
+695
-95
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| |||
0 commit comments