Skip to content
This repository was archived by the owner on Sep 2, 2025. It is now read-only.

Commit a09a8fa

Browse files
use "direct" write for non-partitioned python model materializations (#1388)
* use dynamic schema in test_grant_access_to.py * use dynamic schema in test_grant_access_to.py * revert setup * use "direct" write for non-partitioned python model materializations * add changie log * add code comment * make code comment inline * make code comment inline * remove code comment * use set write_method instead of inline conditional * use set write_method instead of inline conditional
1 parent 79fbd90 commit a09a8fa

File tree

2 files changed

+16
-1
lines changed

2 files changed

+16
-1
lines changed
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
kind: Fixes
2+
body: use "direct" write for non-partitioned python model materializations
3+
time: 2024-10-28T17:27:19.306348-07:00
4+
custom:
5+
Author: colin-rogers-dbt
6+
Issue: "1318"

dbt/include/bigquery/macros/materializations/table.sql

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,10 +113,19 @@ else:
113113
msg = f"{type(df)} is not a supported type for dbt Python materialization"
114114
raise Exception(msg)
115115

116+
# For writeMethod we need to use "indirect" if materializing a partitioned table
117+
# otherwise we can use "direct". Note that indirect will fail if the GCS bucket has a retention policy set on it.
118+
{%- if partition_config %}
119+
{%- set write_method = 'indirect' -%}
120+
{%- else %}
121+
{% set write_method = 'direct' -%}
122+
{%- endif %}
123+
116124
df.write \
117125
.mode("overwrite") \
118126
.format("bigquery") \
119-
.option("writeMethod", "indirect").option("writeDisposition", 'WRITE_TRUNCATE') \
127+
.option("writeMethod", "{{ write_method }}") \
128+
.option("writeDisposition", 'WRITE_TRUNCATE') \
120129
{%- if partition_config is not none %}
121130
{%- if partition_config.data_type | lower in ('date','timestamp','datetime') %}
122131
.option("partitionField", "{{- partition_config.field -}}") \

0 commit comments

Comments
 (0)