Skip to content

Commit a98db2e

Browse files
committed
snowflake_loader: Major improvements with state management
- Add Snowflake-backed persistent state store (amp_stream_state table) - Implement SnowflakeStreamStateStore with overlap detection - Support multiple loading methods: stage, insert, pandas, snowpipe_streaming - Add connection pooling for parallel workers - Implement reorg history tracking with simplified schema - Support Parquet stage loading for better performance State management features: - Block-level overlap detection for different partition sizes - MERGE-based upsert to prevent duplicate state entries - Resume position calculation with gap detection - Deduplication across runs Performance improvements: - Parallel stage loading with connection pool - Optimized Parquet format for stage loads - Efficient batch processing with metadata columns
1 parent 070758a commit a98db2e

File tree

1 file changed

+1655
-110
lines changed

1 file changed

+1655
-110
lines changed

0 commit comments

Comments
 (0)