[Feature] Native BigQuery dataset replication configuration support #1396 #1397
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@gshank - can u take a look? - first PR into dbt repos --> u had done the BigQuery Grants Configurations |
gavin-burns-US
changed the title
add native BigQuery dataset replication configuration support to dbt, allowing users to configure replicas directly in dbt_project.yml, schema files, or model configs without custom hooks. Add changelog for BigQuery dataset replication feature (#5) * Add changelog for BigQuery dataset replication feature * Delete dbt-bigquery/.changes/unreleased/Features-20251015-174021.yaml * Delete dbt-bigquery/.changes/unreleased/Features-20251015-174436.yaml * Delete dbt-bigquery/.changes/unreleased/Fixes-20251009-141523.yaml Update Features-20251015-174512.yaml
gavin-burns-US
force-pushed
the
feature/bigquery/dataset-replication-config
branch
from
cce4e69 to
c792716
Compare
add native BigQuery dataset replication configuration support to dbt, allowing users to configure replicas directly in dbt_project.yml, schema files, or model configs without custom hooks. Add changelog for BigQuery dataset replication feature (#5) * Add changelog for BigQuery dataset replication feature * Delete dbt-bigquery/.changes/unreleased/Features-20251015-174021.yaml * Delete dbt-bigquery/.changes/unreleased/Features-20251015-174436.yaml * Delete dbt-bigquery/.changes/unreleased/Fixes-20251009-141523.yaml Update Features-20251015-174512.yaml
Enhance the description of BigQuery dataset replication configuration support to specify it as a new configuration.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Feature] Native BigQuery dataset replication configuration support #1396
Problem
allowing users to specify replica locations and a primary replica when creating datasets. The changes introduce new configuration options, logic for applying replication settings, and validation to ensure correct usage. Additionally, unit tests are added to verify the new replication logic.
Solution
This feature adds native support for configuring BigQuery dataset replication directly in dbt. Users will be able to specify dataset replicas and a primary replica in dbt_project.yml, schema files, or model configs. This removes the need for custom hooks and enables seamless configuration and management of cross-region dataset replicas within dbt workflows. Implementation includes:
dataset_replicas and primary_replica config options in BigqueryConfig (example)
New create_dataset_with_replication method in BigQueryAdapter (see)
Updated create_dataset method in connections.py (see)
Helper functions in dataset.py for replication logic (see)
Macro for schema creation (see)
Unit tests for replication logic (see).
Checklist
Extras
BigQuery-Dataset-Replication-Configuration-Example.md
pr-summary.md