Skip to content

feat: Allow INDEX syntax sugar for projections with _part_offset - #651

Open
29antonioac wants to merge 11 commits into
ClickHouse:mainfrom
29antonioac:feat/index-syntax-projections
Open

feat: Allow INDEX syntax sugar for projections with _part_offset#651
29antonioac wants to merge 11 commits into
ClickHouse:mainfrom
29antonioac:feat/index-syntax-projections

Conversation

@29antonioac

@29antonioac 29antonioac commented May 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add index parameter to projections model config to allow setting index projections nicer.
  • Using ADD PROJECTION my_projection INDEX my, columns TYPE basic if ClickHouse 26.1+
  • Using ADD PROJECTION my_projection (SELECT _part_offset ORDER BY my, columns) if ClickHouse 25.6+
  • Raise error if ClickHouse <25.6
  • Black has formatted the modified files, modifying a few bits in unmodified functions.
  • Fixes feat: Allow INDEX syntax to add projections with _part_offset from 26.1 #650

Checklist

Delete items not relevant to your PR:


Note

Low Risk
Scoped to projection DDL generation in table materialization with backward-compatible query behavior and compile-time guards; main risk is wrong version-specific SQL on edge ClickHouse versions.

Overview
Adds an optional index field on each entry in the projections model config so you can declare lightweight index projections without writing raw ADD PROJECTION SQL. The add_index_and_projections macro still emits ADD PROJECTION … (query) when query is set; with index only, it builds version-specific DDL: INDEX … TYPE basic on ClickHouse 26.1+, or (SELECT _part_offset ORDER BY …) on 25.6–26.0. Compile-time errors block query + index, missing both, or index on servers before 25.6. The 1.10.1 changelog documents the feature; integration tests cover happy paths, validation messages, and version gating (existing projection tests were reformatted).

Reviewed by Cursor Bugbot for commit 06962eb. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread tests/integration/adapter/projections/test_projections.py
Comment thread dbt/include/clickhouse/macros/materializations/table.sql Outdated
Comment thread tests/integration/adapter/projections/test_projections.py Outdated
Comment thread tests/integration/adapter/projections/test_projections.py
@koletzilla
koletzilla force-pushed the feat/index-syntax-projections branch from 5a783bd to 32c0d34 Compare August 7, 2026 15:22
Comment thread dbt/include/clickhouse/macros/materializations/table.sql Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit bc00721. Configure here.

{%- if not adapter.is_before_version('26.1.1.1') -%}
PROJECTION {{ proj_name }} INDEX {{ cols_index_expr }} TYPE basic
{%- else -%}
PROJECTION {{ proj_name }} (SELECT _part_offset ORDER BY {{ cols_str }})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing ClickHouse 25.6 guard

High Severity

A refactor removed the compile-time check that prevented index projections from being used on ClickHouse versions before 25.6. Now, dbt emits _part_offset-based projection DDL for these older versions, which results in a runtime ClickHouse error. This failure happens later in the run, potentially after distributed materializations have already dropped tables.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit bc00721. Configure here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed, version not supported.

@koletzilla

Copy link
Copy Markdown
Contributor

Hi @29antonioac , I did a few changes to the PR, can you do a second check of them?

  • I have moved the validation also to the start of these materializations. As the materializations do drops, It's better if we detect these problems before tables are changed.
  • I have removed the keyword argument: it adds a bit of noise in terms of configuration and in the end we need to include it somehow. So now the function just writes the PROJECTION part and that's all.
  • I have removed the check for 25.6. We officially don't support this version and it's not tested in the CI. In any case CH will raise this error if the table is created with that, so somehow these versions "are still covered by some error information".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Allow INDEX syntax to add projections with _part_offset from 26.1

2 participants