This repository was archived by the owner on Sep 2, 2025. It is now read-only.
dbt copy_partitions with job parallelization in Bigquery #1236
Unanswered
carolinabtt
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I am creating dbt models with copy_partitions=true, but the problem is when I need to ingest a number of partitions with a large volume of data, the whole process can take a long time as dbt iterates by calling the Bigquery API for each partition.
Is there a way to parallelize those copy_partitions jobs in dbt?
I tried Threads configuration but does not work, since threads parallelize models executions and not API calls.
Beta Was this translation helpful? Give feedback.
All reactions