Commit cf223d4
committed
feat: Major improvements to Snowflake loader with state management
- Add Snowflake-backed persistent state store (amp_stream_state table)
- Implement SnowflakeStreamStateStore with overlap detection
- Support multiple loading methods: stage, insert, pandas,
snowpipe_streaming
- Add connection pooling for parallel workers
- Implement reorg history tracking with simplified schema
- Support Parquet stage loading for better performance
State management features:
- Block-level overlap detection for different partition sizes
- MERGE-based upsert to prevent duplicate state entries
- Resume position calculation with gap detection
- Deduplication across runs
Performance improvements:
- Parallel stage loading with connection pool
- Optimized Parquet format for stage loads
- Efficient batch processing with metadata columns1 parent 92151df commit cf223d4
1 file changed
+1655
-110
lines changed
0 commit comments