Skip to content

Comments

feat: Enable concurrent microbatch execution#1326

Open
wmjones wants to merge 2 commits intodatabricks:mainfrom
wmjones:914-microbatch-concurrency
Open

feat: Enable concurrent microbatch execution#1326
wmjones wants to merge 2 commits intodatabricks:mainfrom
wmjones:914-microbatch-concurrency

Conversation

@wmjones
Copy link

@wmjones wmjones commented Feb 18, 2026

Resolves #914

Description

Declares MicrobatchConcurrency adapter capability so dbt-core 1.9+ can execute microbatch incremental batches in parallel threads instead of sequentially.

This is a one-line addition to DatabricksAdapter._capabilities. dbt-core already handles all concurrency orchestration — temp table uniqueness (via model.batch.id suffix in make_temp_relation), non-overlapping REPLACE WHERE predicates per batch, and sequential execution of first/last batches. The adapter just needs to opt in.

Why Support.Full: Concurrency control happens at dbt-core's Python threading level, not Databricks SQL level. Every DBR version and SQL Warehouse that supports replace_where inherently supports concurrent batch execution.

OPTIMIZE note: Per-batch OPTIMIZE calls can conflict under concurrency when liquid_clustered_by, zorder, or auto_liquid_cluster is configured. Users with those configs can set DATABRICKS_SKIP_OPTIMIZE=true as a workaround. Plain Delta tables are unaffected.

Tested on: SQL Warehouse with Unity Catalog (30-batch microbatch model, --threads 4, all batches passed, concurrent execution confirmed, zero duplicates).

Prior art: dbt-snowflake added the same capability in dbt-snowflake#1259.

Checklist

  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

@wmjones wmjones force-pushed the 914-microbatch-concurrency branch from 3d91722 to 060f1b9 Compare February 19, 2026 13:23
@sd-db
Copy link
Collaborator

sd-db commented Feb 19, 2026

Hi @wmjones, we are doing a the release today for 1.11.5 and would not be possible to add a new feature in the same release (includes dbt-core pinning upgrade). Can you update the CHANGELOG.md to target 1.11.5 instead ? Thx again for the PR !!

P.S. I am stating integration test runs and will report if I see any issues.

…y capability

Signed-off-by: Wyatt Jones <wyatt.jones6@cfacorp.com>
@wmjones wmjones force-pushed the 914-microbatch-concurrency branch from 060f1b9 to 04e0846 Compare February 20, 2026 00:02
@sd-db
Copy link
Collaborator

sd-db commented Feb 24, 2026

@wmjones It seem concurrency is opt-in by default. What this means is that anyone who is using microbatch-incremental models will have concurrent execution enabled by default. In dbt-databricks there is a well-established behaviour flag system that you can use to gate this feature behind a flag (it would be False by default). I would suggest doing that so it can ship safely in a patch, and then flip the default to true around 1.12 as we should have good data points by then on prod behaviour. See --> b819a878 for more details on how you might do this...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support concurrent_batches for microbatch

2 participants