Skip to content

Commit ac0d194

Browse files
committed
Add BigQuery relation naming macros for concurrent execution
Implements BigQuery-specific relation naming macros to enable concurrent dbt execution without relation name collisions, matching functionality available in PostgreSQL adapter. Added macros: - bigquery__make_relation_with_suffix: Core macro with truncation and datetime - bigquery__make_temp_relation: Temp relations with unique datetime suffixes - bigquery__make_intermediate_relation: Intermediate relations without datetime - bigquery__make_backup_relation: Backup relations with configurable types
1 parent 80b5057 commit ac0d194

File tree

1 file changed

+31
-0
lines changed

1 file changed

+31
-0
lines changed

dbt-bigquery/src/dbt/include/bigquery/macros/adapters.sql

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,3 +209,34 @@ having count(*) > 1
209209
{% do adapter.upload_file(local_file_path, database, table_schema, table_name, kwargs=kwargs) %}
210210

211211
{% endmacro %}
212+
213+
{% macro bigquery__make_relation_with_suffix(base_relation, suffix, dstring) %}
214+
{% if dstring %}
215+
{% set dt = modules.datetime.datetime.now() %}
216+
{% set dtstring = dt.strftime("%j%H%M%S") %}
217+
{% set suffix = suffix ~ "_" ~ dtstring %}
218+
{% endif %}
219+
{% set suffix_length = suffix|length %}
220+
{% set relation_max_name_length = 1024 %} {# BigQuery limit #}
221+
{% if suffix_length > relation_max_name_length %}
222+
{% do exceptions.raise_compiler_error('Relation suffix is too long (' ~ suffix_length ~ ' characters). Maximum length is ' ~ relation_max_name_length ~ ' characters.') %}
223+
{% endif %}
224+
{% set identifier = base_relation.identifier[:relation_max_name_length - suffix_length] ~ suffix %}
225+
226+
{{ return(base_relation.incorporate(path={"identifier": identifier })) }}
227+
228+
{% endmacro %}
229+
230+
{% macro bigquery__make_temp_relation(base_relation, suffix) %}
231+
{% set temp_relation = bigquery__make_relation_with_suffix(base_relation, suffix, dstring=True) %}
232+
{{ return(temp_relation) }}
233+
{% endmacro %}
234+
235+
{% macro bigquery__make_intermediate_relation(base_relation, suffix) %}
236+
{{ return(bigquery__make_relation_with_suffix(base_relation, suffix, dstring=False)) }}
237+
{% endmacro %}
238+
239+
{% macro bigquery__make_backup_relation(base_relation, backup_relation_type, suffix) %}
240+
{% set backup_relation = bigquery__make_relation_with_suffix(base_relation, suffix, dstring=False) %}
241+
{{ return(backup_relation.incorporate(type=backup_relation_type)) }}
242+
{% endmacro %}

0 commit comments

Comments
 (0)