- 
                Notifications
    You must be signed in to change notification settings 
- Fork 208
Open
Open
Copy link
Labels
feature:incrementalIssues related to incremental materializationsIssues related to incremental materializationspkg:dbt-bigqueryIssue affects dbt-bigqueryIssue affects dbt-bigquerytype:enhancementNew feature requestNew feature request
Description
Is this your first time submitting a feature request?
- I have read the expectations for open source contributors
- I have searched the existing issues, and I could not find an existing issue for this feature
- I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion
Describe the feature
The MERGE statement is sub-optimized in BigQuery.
This is of course ok for unique keys as this is what we are looking for.
But for the insert_overwrite strategy where we are looking to rows at the partition-level there is a better solution and here is why:
- a DELETEorINSERTstatement is cheapest than aMERGEstatement.
- incremental tables are the most expensive tables in real-world projects.
- The DELETEstatement in BigQuery is free at the partition-level.
This has been tested at Carrefour which is my company.
- On this replacement of the MERGEstatement it reduces the cost by 50.4% and the elapsed time by 35.2%
- On the overall procedure it reduces the cost by 26.1% and the elapsed time by 23.1%
This is wrapped in a transaction to avoid deleting rows if any error occurs.
Describe alternatives you've considered
I have considered an alternative to create an additional delete+insert incremental strategy.
But overriding the existing insert_overwrite is the more convenient as this is exactly what we are looking for and the features are the same.
Who will this benefit?
Of course, everyone without any changes.
Are you interested in contributing this feature?
Yes
Anything else?
No response
Liscor, benedikt-buchert, AxelThevenot and PrettyMaryK
Metadata
Metadata
Assignees
Labels
feature:incrementalIssues related to incremental materializationsIssues related to incremental materializationspkg:dbt-bigqueryIssue affects dbt-bigqueryIssue affects dbt-bigquerytype:enhancementNew feature requestNew feature request