Skip to content

Commit 8795478

Browse files
benc-dbericj-dbclaudeKorijnQMalcolm
authored
chore: Making 1.11.0 as main dev in prep for release (#1242)
### Description Merging 1.11.latest to main to begin 1.11.0 release ### Checklist - [x] I have run this code in development and it appears to resolve the stated issue - [x] This PR includes tests, or tests are not required/relevant for this PR - [x] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-databricks next" section. --------- Signed-off-by: Korijn van Golen <[email protected]> Signed-off-by: Randy Pitcher <[email protected]> Signed-off-by: Can Bekleyici <[email protected]> Signed-off-by: Adam Staros <[email protected]> Co-authored-by: Eric Jang <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Korijn van Golen <[email protected]> Co-authored-by: Quigley Malcolm <[email protected]> Co-authored-by: Quigley Malcolm <[email protected]> Co-authored-by: Sam Chou <[email protected]> Co-authored-by: Eric Jang <[email protected]> Co-authored-by: Thang Long Vu <[email protected]> Co-authored-by: Randy Pitcher <[email protected]> Co-authored-by: Can Bekleyici <[email protected]> Co-authored-by: adam-staros95 <[email protected]> Co-authored-by: astaros <[email protected]>
1 parent f029291 commit 8795478

File tree

129 files changed

+5860
-1432
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

129 files changed

+5860
-1432
lines changed

.github/workflows/integration.yml

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,12 +68,15 @@ jobs:
6868
id: setup-python
6969
uses: actions/setup-python@v5
7070
with:
71-
python-version: "3.9"
71+
python-version: "3.10"
7272

7373
- name: Get http path from environment
7474
run: python .github/workflows/build_cluster_http_path.py
7575
shell: sh
7676

77+
- name: Install uv
78+
uses: astral-sh/setup-uv@v4
79+
7780
- name: Install Hatch
7881
id: install-dependencies
7982
uses: pypa/hatch@install
@@ -118,12 +121,15 @@ jobs:
118121
id: setup-python
119122
uses: actions/setup-python@v5
120123
with:
121-
python-version: "3.9"
124+
python-version: "3.10"
122125

123126
- name: Get http path from environment
124127
run: python .github/workflows/build_cluster_http_path.py
125128
shell: sh
126129

130+
- name: Install uv
131+
uses: astral-sh/setup-uv@v4
132+
127133
- name: Install Hatch
128134
id: install-dependencies
129135
uses: pypa/hatch@install
@@ -165,12 +171,15 @@ jobs:
165171
id: setup-python
166172
uses: actions/setup-python@v5
167173
with:
168-
python-version: "3.9"
174+
python-version: "3.10"
169175

170176
- name: Get http path from environment
171177
run: python .github/workflows/build_cluster_http_path.py
172178
shell: sh
173179

180+
- name: Install uv
181+
uses: astral-sh/setup-uv@v4
182+
174183
- name: Install Hatch
175184
id: install-dependencies
176185
uses: pypa/hatch@install

.github/workflows/main.yml

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,10 @@ jobs:
5757
- name: Set up Python
5858
uses: actions/setup-python@v5
5959
with:
60-
python-version: "3.9"
60+
python-version: "3.10"
61+
62+
- name: Install uv
63+
uses: astral-sh/setup-uv@v4
6164

6265
- name: Install Hatch
6366
uses: pypa/hatch@install
@@ -83,7 +86,7 @@ jobs:
8386
strategy:
8487
fail-fast: false
8588
matrix:
86-
python-version: ["3.9", "3.10", "3.11", "3.12"]
89+
python-version: ["3.10", "3.11", "3.12", "3.13"]
8790

8891
steps:
8992
- name: Check out the repository
@@ -94,6 +97,9 @@ jobs:
9497
with:
9598
python-version: ${{ matrix.python-version }}
9699

100+
- name: Install uv
101+
uses: astral-sh/setup-uv@v4
102+
97103
- name: Install Hatch
98104
uses: pypa/hatch@install
99105

@@ -125,7 +131,10 @@ jobs:
125131
- name: Set up Python
126132
uses: actions/setup-python@v5
127133
with:
128-
python-version: "3.9"
134+
python-version: "3.10"
135+
136+
- name: Install uv
137+
uses: astral-sh/setup-uv@v4
129138

130139
- name: Install Hatch
131140
uses: pypa/hatch@install

.pre-commit-config.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ repos:
55
hooks:
66
# Run the linter.
77
- id: ruff
8+
args: [--fix]
89
# Run the formatter.
910
- id: ruff-format
1011

AGENTS.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ This guide helps AI agents quickly understand and work productively with the dbt
99
- **What**: dbt adapter for Databricks Lakehouse platform
1010
- **Based on**: dbt-spark adapter with Databricks-specific enhancements
1111
- **Key Features**: Unity Catalog support, Delta Lake, Python models, streaming tables
12-
- **Language**: Python 3.9+ with Jinja2 SQL macros
12+
- **Language**: Python 3.10+ with Jinja2 SQL macros
1313
- **Architecture**: Inherits from Spark adapter, extends with Databricks-specific functionality
1414

1515
### Essential Files to Understand
@@ -34,7 +34,7 @@ dbt/include/databricks/macros/ # Jinja2 SQL templates
3434

3535
## 🛠 Development Environment
3636

37-
**Prerequisites**: Python 3.9+ installed on your system
37+
**Prerequisites**: Python 3.10+ installed on your system
3838

3939
**Install Hatch** (recommended):
4040

CHANGELOG.md

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,44 @@
1-
## dbt-databricks 1.10.15 (TBD)
1+
## dbt-databricks 1.11.1 (TBD)
2+
3+
## dbt-databricks 1.11.0 (Oct 29, 2025)
4+
5+
### Features
6+
7+
- Support databricks_tags for MV/STs
8+
- Add support for scalar SQL functions (SQL UDFs) ([1197](https://github.com/databricks/dbt-databricks/pull/1197))
9+
- Add liquid clustering config for materialized views and streaming tables (thanks @reflection!) ([1101](https://github.com/databricks/dbt-databricks/pull/1101))
10+
- Add official support for `insert_overwrite` incremental strategy for SQL warehouses. This strategy now uses `REPLACE ON` syntax for all compute types (previously `INSERT OVERWRITE`). This behavior is gated behind behavior flag `use_replace_on_for_insert_overwrite` which default `true` ([1025](https://github.com/databricks/dbt-databricks/issues/1025))
11+
- Add support for Databricks query tags
12+
- Add support for managed iceberg when `table_format` is set to `iceberg`. This behavior is gated behind behavior flag `use_managed_iceberg` which defaults to `false`
13+
- Support delete+insert incremental strategy (thanks @canbekley!) ([1217](https://github.com/databricks/dbt-databricks/issues/1217))
14+
15+
### Fixes
16+
17+
- **BREAKING:** Fix column order mismatch bug in incremental models by using INSERT BY NAME syntax ([#1211](https://github.com/databricks/dbt-databricks/issues/1211))
18+
- When using `on_schema_change: sync_all_columns`, dbt previously used positional column matching in INSERT statements, causing values to be inserted into wrong columns when column order changed
19+
- Now uses Databricks `INSERT BY NAME` syntax to match columns by name instead of position, preventing data corruption
20+
- **Breaking Change**: Requires Databricks Runtime 12.2 LTS or higher
21+
- Users on older runtimes should pin to dbt-databricks 1.10.x
22+
- Affects all incremental strategies: `append`, `insert_overwrite`, `replace_where`, and `merge` (via table creation)
23+
- Fix case-sensitivity issues with column name handling in persist_docs and config diff operations ([#1215](https://github.com/databricks/dbt-databricks/issues/1215))
24+
- Fixed KeyError when column names in models had different casing than YAML schema definitions
25+
- Improved efficiency of column tags and comments change detection to use case-insensitive comparison
26+
- Use backtick quoting for everything to avoid errors with special characters ([1186](https://github.com/databricks/dbt-databricks/pull/1186))
27+
- Ensure column compare always uses lower case names (since Databricks stores internally as lower case) ([1190](https://github.com/databricks/dbt-databricks/pull/1190))
28+
- Fix incompatible schema error during streaming table creation ([1235](https://github.com/databricks/dbt-databricks/issues/1235))
29+
- Reintroduce support for external so as not to break users ([1240](1240))
30+
31+
### Under the Hood
32+
33+
- Materialized views now uses `CREATE OR REPLACE` where appropriate, instead of DROP + CREATE
34+
- Refactor to use Databricks SDK for API calls ([1185](https://github.com/databricks/dbt-databricks/pull/1185))
35+
- Update dependency versions, and start using uv ([1199](https://github.com/databricks/dbt-databricks/pull/1199))
36+
- Upgrade ruff and mypy ([1207](https://github.com/databricks/dbt-databricks/pull/1207))
37+
- Allow create or replace semantics on full refresh in Mat V2 ([1210](https://github.com/databricks/dbt-databricks/pull/1210))
38+
- Add centralized DBR capability system for managing version-dependent features with per-compute caching ([#1218](https://github.com/databricks/dbt-databricks/pull/1218))
39+
- **BREAKING:** Removing the 'use_info_schema_for_columns' behavior flag, as we have a better mechanism for getting complex type information - DESCRIBE EXTENDED ... AS JSON. This is a breaking change because it requires a modern DBR (or SQL Warehouse) in order to function ([1226](https://github.com/databricks/dbt-databricks/pull/1226))
40+
- Use atomic `CREATE OR REPLACE` instead of DROP + CREATE for managed Iceberg tables
41+
- Drop support for python 3.9, adds 3.13 ([1240](https://github.com/databricks/dbt-databricks/pull/1240))
242

343
## dbt-databricks 1.10.14 (October 22, 2025)
444

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
version = "1.10.14"
1+
version = "1.11.0"

0 commit comments

Comments
 (0)