-
Notifications
You must be signed in to change notification settings - Fork 207
Description
Is this your first time submitting a feature request?
- I have read the expectations for open source contributors
- I have searched the existing issues, and I could not find an existing issue for this feature
- I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
Describe the feature
This feature can improve the speed of UPDATE, DELETE, and MERGE queries which could be helpful with incremental dbt model with the merge strategy as well as potential implementations of GDPR processes.
As they say themselves:
When enabled, instead of always rewriting large storage blocks, BigQuery fine-grained DML can pinpoint and modify data with much finer granularity. It leverages optimized metadata indexes to rewrite only the necessary mutated data, drastically reducing the processing, I/O, and consequently, the slot time consumed for sparse DML. The result? Faster, more cost-effective DM
Describe alternatives you've considered
I considered including it in exactly the same way as the enable_change_history in this PR: #1225
These features are both described in the blog.
Who will this benefit?
Incremental merge queries could see increased performance.
GDPR implementations that anonymize or delete data in tables created by dbt.
Are you interested in contributing this feature?
Yes. I have adapted the #1225 for this option
Anything else?
https://cloud.google.com/blog/products/data-analytics/bigquery-features-for-transactional-data-management
https://cloud.google.com/bigquery/docs/data-manipulation-language#fine-grained_dml