Skip to content

Commit 70689cb

Browse files
committed
*: Major base loader improvements for streaming and resilience
- Integrate state management for resume and deduplication - Add label joining support with automatic type conversion - Implement resilience features (retry, backpressure, rate limiting) - Add metadata columns (_amp_batch_id) for reorg handling - Support streaming with block ranges and reorg detection - Separate _try_load_batch() for better error handling
1 parent 220b7b9 commit 70689cb

File tree

5 files changed

+695
-95
lines changed

5 files changed

+695
-95
lines changed

src/amp/loaders/__init__.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323

2424
from .base import DataLoader
2525
from .registry import LoaderRegistry, create_loader, get_available_loaders, get_loader_class
26-
from .types import LoadConfig, LoadMode, LoadResult
26+
from .types import LabelJoinConfig, LoadConfig, LoadMode, LoadResult
2727

2828
# Trigger auto-discovery on import
2929
LoaderRegistry._ensure_auto_discovery()
@@ -32,6 +32,7 @@
3232
'DataLoader',
3333
'LoadResult',
3434
'LoadConfig',
35+
'LabelJoinConfig',
3536
'LoadMode',
3637
'LoaderRegistry',
3738
'get_loader_class',

0 commit comments

Comments
 (0)