[destination-snowflake] configurable batch sizes and flush frequency #35249
-
TopicMake snowflake batch sizes and flush frequency configurable Relevant informationIssue 31741 was opened to increase the size at which snowflake flushes from 10mb to 200mb More frequent flushes are great for making sure that data gets to snowflake as quickly as possible; however, they significantly increase snowflake costs on slower sources. If we could configure the snowflake connector flush size and maximum time between flushes it would allow users to tailor the setting to prioritize performance vs cost depending on the use case. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
We aren't going to be working on this in the near future, so I'm closing this issue. The confounding principle is checkpointing - Airbyte wants to commit data at least once every 15 minutes. So the upper bound of batch-size is 200mb (snowflake's recommendation), and the lower bound is "however much data the stream gets in 15 min". |
Beta Was this translation helpful? Give feedback.
We aren't going to be working on this in the near future, so I'm closing this issue. The confounding principle is checkpointing - Airbyte wants to commit data at least once every 15 minutes. So the upper bound of batch-size is 200mb (snowflake's recommendation), and the lower bound is "however much data the stream gets in 15 min".