Custom dbt Transformations: Add method to supply custom params to profiles.yml output

[jfillmore]

What area the feature impact?

Connectors

Revelant Information

We are using Airbyte with DBT and BigQuery as the destination. We're having frequent problems with connection runs stalling out because DBT gets stuck waiting on BigQuery API calls that never complete (verified using lsof/strace/gdb to pinpoint precisely). This causes the run to hang until SYNC_JOB_MAX_TIMEOUT_DAYS is reached, which we need to be relatively large to allow for ad hoc large backfills, source data selection growth, and

DBT has the configuration option https://docs.getdbt.com/docs/core/connect-data-platform/bigquery-setup#job_execution_timeout_seconds that seems the best way for us to address this issue, but there seems to be no way to set this within Airbyte when entrypoint.sh calls transform-config to generate profiles.yml. Because this configuration option is specific to the profile (e.g. "prod" in Airbyte's case), there aren't any env variables for CLI options available within DBT that can be set.

---

We are currently solving this by using a custom DBT image that hijacks /usr/local/bin/dbt to rewrite profiles.yml on-the-fly to suit our needs, but because there are a lot of potentially useful options we think it would be great to have a way to set parameters dynamically within the custom transformation configuration.

Some ideas:

1. Add an additional configuration field to specify a path to a YML file packaged within your DBT repo that contains the parameters.
2. Add an additional configuration field to supply the parameters as JSON values, similar to how DBT entrypoint arguments are provided.

Thanks for your consideration!

[marcosmarxm]

Thanks for the proposition @jfillmore I moved to become a discussion and receive votes.

[jfillmore]

One more comment to make about this issue. We continue to be plagued by problems caused by DBT defaulting to infinite timeouts for some HTTP requests that aren't respecting it's own timeout options.

Perhaps a simpler, more direct approach is just to expose an option within Airbyte to customize how long the normalization step is allowed to take? I think this would compliment the global SYNC_JOB_MAX_TIMEOUT_DAYS option nicely to allow more fine-grained control.

Internally, we've been forced to apply an additional hack, much like the one described above. Our custom Docker image for DBT provides a wrapper around /usr/local/bin/dbt now forks off a sleep thread in the background to prematurely kill long-running jobs.