Skip to content

[Feature] Add enable_fine_grained_mutations support BigQuery #1386

@Thrasi

Description

@Thrasi

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

This feature can improve the speed of UPDATE, DELETE, and MERGE queries which could be helpful with incremental dbt model with the merge strategy as well as potential implementations of GDPR processes.
As they say themselves:

When enabled, instead of always rewriting large storage blocks, BigQuery fine-grained DML can pinpoint and modify data with much finer granularity. It leverages optimized metadata indexes to rewrite only the necessary mutated data, drastically reducing the processing, I/O, and consequently, the slot time consumed for sparse DML. The result? Faster, more cost-effective DM

Describe alternatives you've considered

I considered including it in exactly the same way as the enable_change_history in this PR: #1225
These features are both described in the blog.

Who will this benefit?

Incremental merge queries could see increased performance.
GDPR implementations that anonymize or delete data in tables created by dbt.

Are you interested in contributing this feature?

Yes. I have adapted the #1225 for this option

Anything else?

https://cloud.google.com/blog/products/data-analytics/bigquery-features-for-transactional-data-management
https://cloud.google.com/bigquery/docs/data-manipulation-language#fine-grained_dml

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions