Skip to content

Commit cf223d4

Browse files
committed
feat: Major improvements to Snowflake loader with state management
- Add Snowflake-backed persistent state store (amp_stream_state table) - Implement SnowflakeStreamStateStore with overlap detection - Support multiple loading methods: stage, insert, pandas, snowpipe_streaming - Add connection pooling for parallel workers - Implement reorg history tracking with simplified schema - Support Parquet stage loading for better performance State management features: - Block-level overlap detection for different partition sizes - MERGE-based upsert to prevent duplicate state entries - Resume position calculation with gap detection - Deduplication across runs Performance improvements: - Parallel stage loading with connection pool - Optimized Parquet format for stage loads - Efficient batch processing with metadata columns
1 parent 92151df commit cf223d4

File tree

1 file changed

+1655
-110
lines changed

1 file changed

+1655
-110
lines changed

0 commit comments

Comments
 (0)