DE-8600: DBT support for all pipeline & output stream configurations

[jbreeden]

This is a new, squashed, PR for the draft that was opened a few weeks ago. The draft was a mess with my Python skill issues.

The general idea of this PR is to change the config schema to take a partial Decodable Resource Spec for each of these objects, rather than just accepting some subset of those specs directly.

The partial spec from the config is then merged with the bits that DBT usually infers, such as the fields (inferred from the SQL), the names, etc.

Then this complete spec is sent to our new /apply endpoint. This endpoint has a lot of improvements over our older APIs, and has exclusive access to some newer features. It also avoids the requirement of stopping jobs manually before applying updates, which simplifies the adapter code a bit, and sometimes makes for a better UX.

See the http_events example to get an idea of how this would change the interface.

Note on the approach: My hope is to move the DBT adapter to being a somewhat simple layer on top of our Declarative API features. A consequence of this is that you'll see me handling the resoource specs as basically just YAML (or the python Dict representation of that YAML). This would let us add fields to the API, and have them picked up by DBT automatically without publishing a new version. The current code is not agnostic to the spec versions, so picking up new versions of our resource spec would require a new adapter version. My gut feeling was that this is a decent compromise of flexibility & convenience, but I'd love to hear from others.

We've also discussed spitting out Decodable Declarative Resource YAML from the dbt compile step. With this done, I think that amounts to basically spitting out the spec we build from this configuration. There may be some "details", but this work should serve both.

Evidence of working examples

Here's a run of the examples after this change. There's not real change to the output, but at least you can see they work. I've also tested making changes to the example configurations and ensured that the right resources are recreated as expected.

❯ DECODABLE_PROFILE=jbreeden dbt run
15:55:07  Running with dbt=1.3.2
15:55:07  Found 3 models, 0 tests, 0 snapshots, 0 analyses, 273 macros, 0 operations, 1 seed file, 0 sources, 0 exposures, 0 metrics
15:55:07
15:55:08  Concurrency: 1 threads (target='dev')
15:55:08
15:55:08  1 of 3 START sql table model dbt__events_count ................................. [RUN]
15:55:09  1 of 3 OK created sql table model dbt__events_count ............................ [OK in 1.26s]
15:55:09  2 of 3 START sql table model dbt__http_events .................................. [RUN]
15:55:10  2 of 3 OK created sql table model dbt__http_events ............................. [OK in 1.32s]
15:55:10  3 of 3 START sql table model dbt__http_events_bytes_sent ....................... [RUN]
15:55:12  3 of 3 OK created sql table model dbt__http_events_bytes_sent .................. [OK in 1.24s]
15:55:12
15:55:12  Finished running 3 table models in 0 hours 0 minutes and 4.45 seconds (4.45s).
15:55:12
15:55:12  Completed successfully
15:55:12
15:55:12  Done. PASS=3 WARN=0 ERROR=0 SKIP=0 TOTAL=3

[enthal]

Note/FYI:
From the PR description:

A consequence of this is that you'll see me handling the resoource specs as basically just YAML (or the python Dict representation of that YAML). This would let us add fields to the API, and have them picked up by DBT automatically without publishing a new version.

Yes! It is good.

The current code is not agnostic to the spec versions, so picking up new versions of our resource spec would require a new adapter version.

And that is right and proper: new fields are added without changing the spec version. A new spec version is only made available for something backward incompatible for existing clients, so they can opt in. We've only added a new spec version once: for Declarative Execution, which changed a default.

You prolly know all this, but here it is anyway, for posterity.

[enthal]

LGTM!