Cluster Joins part 2 - global mode by ianton-ru · Pull Request #1527 · Altinity/ClickHouse

ianton-ru · 2026-03-13T15:52:38Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Cluster Joins part 2 - global mode

Documentation entry for user-facing changes

Setting object_storage_cluster_join_mode wiith value global.
In queries like

SELECT * FROM iceberg_table(...) JOIN local_table(...) ON ...

when left table is executed on cluster (s3Cluster, Iceberg with object_storage_cluster setting, etc.) data from right table is extracted and sent to swarm nodes as temorary tables. JOIN is executed on swarm nodes.

CI/CD Options

Exclude tests:

Regression jobs to run:

ianton-ru · 2026-03-13T15:53:04Z

@codex review

github-actions · 2026-03-13T15:53:48Z

Workflow [PR], commit [d0de2cf]

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38e89f4657

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-13T16:09:09Z

src/Storages/IStorageCluster.cpp

+    if (send_external_tables)
+        external_tables = query_info.planner_context->getMutableQueryContext()->getExternalTables();


Reset external-table forwarding state per query

send_external_tables is process state on the storage object and is only ever set to true in the GLOBAL-join path, so once one query enables it, later reads in other modes still take this branch; in those later reads query_info.planner_context may be null (old-analyzer path), so query_info.planner_context->getMutableQueryContext() can crash, and even when non-null it forwards external tables for unrelated queries. This needs to be scoped/reset per read instead of persisting across requests.

Useful? React with 👍 / 👎.

send_external_tables can be true only when object_storage_cluster_join_mode='global'. But if allow_experimental_analyzer=0 with object_storage_cluster_join_mode='global', exception n getQueryProcessingStage must be thrown.

ianton-ru · 2026-03-18T12:14:27Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cb5e474d83

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-18T12:39:00Z

src/Storages/IStorageCluster.h


    LoggerPtr log;
    String cluster_name;
+    bool send_external_tables = false;


Keep external-table send flag scoped to a single read

send_external_tables is stored as mutable state on IStorageCluster, but it is set from one query path and never reset before subsequent reads. After a GLOBAL-mode query sets it, later queries on the same storage instance can still forward external tables even when rewrite is not active; and because this field is non-atomic shared state, concurrent reads can race on it. This creates cross-query state leakage and undefined behavior risk in production read concurrency.

Useful? React with 👍 / 👎.

Is this true?

chatgpt-codex-connector · 2026-03-18T12:39:00Z

src/Storages/IStorageCluster.cpp

            throw Exception(ErrorCodes::NOT_IMPLEMENTED,
-                "object_storage_cluster_join_mode!='allow' is not supported without allow_experimental_analyzer=true");
+                "object_storage_cluster_join_mode!='allow' is not supported without allow_experimental_analyzer=false");


Correct inverted analyzer requirement in error message

This branch is entered when allow_experimental_analyzer is disabled, but the message says the mode is unsupported without allow_experimental_analyzer=false. That inverts the required value and points users to the wrong configuration change when they hit this exception.

Useful? React with 👍 / 👎.

ianton-ru · 2026-03-23T11:17:16Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e3fe1a1c18

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-23T11:30:28Z

src/Storages/IStorageCluster.cpp

+                /*allow_global_join_for_right_table*/ true,
+                /*find_cross_join*/ true);
+            query_to_send = queryNodeToDistributedSelectQuery(modified_query_tree);
+            send_external_tables = true;


Keep external-table state per query, not on storage instance

Setting send_external_tables here stores query-specific state on the shared IStorageCluster object, but that flag is later read/cleared in read(). For storage engines that reuse one instance across requests (for example StorageObjectStorageCluster), concurrent reads can interleave so one query clears the flag while another still needs it, causing the second query to skip sending generated _data_* external tables to shards and fail at execution time. This state needs to be local to a single read path (or otherwise synchronized) rather than kept as mutable object state.

Useful? React with 👍 / 👎.

ianton-ru · 2026-03-23T12:29:12Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0cd90a87ab

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-23T12:48:22Z

src/Storages/buildQueryTreeForShard.cpp

+            bool prefer_local_join = getContext()->getSettingsRef()[Setting::parallel_replicas_prefer_local_join]
+                && getContext()->getSettingsRef()[Setting::object_storage_cluster_join_mode] != ObjectStorageClusterJoinMode::GLOBAL;


Keep local-join preference outside object-storage mode

rewriteJoinToGlobalJoin is also used by parallel-replica planning (src/Planner/findParallelReplicasQuery.cpp and src/Interpreters/ClusterProxy/executeQuery.cpp), so tying prefer_local_join to object_storage_cluster_join_mode here makes unrelated non-object-storage queries ignore parallel_replicas_prefer_local_join whenever the session sets object_storage_cluster_join_mode='global'. That forces extra GLOBAL JOIN rewrites in those flows and can significantly increase broadcast/memory costs; this override should be scoped to the object-storage cluster path instead of the shared visitor.

Useful? React with 👍 / 👎.

Yes, object_storage_cluster_join_mode=global has priority and oveerides parallel_replicas_prefer_local_join setting here.

ianton-ru · 2026-03-23T15:34:20Z

Audit: PR #1527 — Cluster Joins part 2 (global mode)

PR: Altinity/ClickHouse#1527 — Cluster Joins part 2 - global mode

Scope: object_storage_cluster_join_mode='global', object storage cluster reads (IStorageCluster), query rewrite (rewriteJoinToGlobalJoin, buildQueryTreeForShard), ReadFromCluster / RemoteQueryExecutor external tables, settings and tests.

AI audit note: This review was generated by AI (Cursor agent), following .cursor/skills/audit-review/SKILL.md.

Confirmed defects

Low — Stale and misleading documentation for `object_storage_cluster_join_mode`

Impact: Users and operators can believe global is still unsupported or that the setting only applies to JOIN subqueries, while the implementation enables global and applies the rewrite when has_join, has_cross_join, or has_local_columns_in_where is true (see IStorageCluster::updateQueryWithJoinToSendIfNeeded / getQueryTreeInfo).
Anchor: src/Core/Settings.cpp — DECLARE(..., object_storage_cluster_join_mode, ...) help string.
Trigger: Reading built-in setting documentation (e.g. system.settings, docs generated from the same string).
Why defect: The shipped description contradicts behavior and PR scope (global implemented; not “JOIN-only” in code).
Fix direction: Update the R"(...)" text for object_storage_cluster_join_mode to describe global as supported and align “Restrictions” with getQueryTreeInfo / actual rewrite conditions.
Regression test direction: Optional: lightweight check that the help string does not contain “Unsupported for now” for global (if the project tests setting descriptions).

Evidence:

    DECLARE(ObjectStorageClusterJoinMode, object_storage_cluster_join_mode, ObjectStorageClusterJoinMode::ALLOW, R"(
Changes the behaviour of object storage cluster function or table.
...
Restrictions:

- Only applied for JOIN subqueries.
- Only if the FROM section uses a object storage cluster function or table.

Possible values:

- `local` — Replaces the database and table in the subquery with local ones for the destination server (shard), leaving the normal `IN`/`JOIN.`
- `global` — Unsupported for now. Replaces the `IN`/`JOIN` query with `GLOBAL IN`/`GLOBAL JOIN.`
- `allow` — Default value. Allows the use of these types of subqueries.
)", 0) \

Notes (not classified as confirmed code defects)

Earlier Codex concern (mutable send_external_tables on IStorageCluster): In the current tree this is not present; external tables are taken from query_info.planner_context and passed into ReadFromCluster, avoiding cross-query leakage on the storage object.
parallel_replicas_prefer_local_join vs object_storage_cluster_join_mode=global: RewriteJoinToGlobalJoinVisitor in buildQueryTreeForShard.cpp intentionally gives global priority over local join preference; no incorrect-result path was verified in static review.

Coverage summary

Item	Detail
Scope reviewed	End-to-end path for global object-storage cluster joins: `getQueryProcessingStage` (analyzer gate), `updateQueryWithJoinToSendIfNeeded` (GLOBAL branch), `read` → `ReadFromCluster` → `RemoteQueryExecutor` with `external_tables`; visitors in `buildQueryTreeForShard.cpp`; integration tests in `tests/integration/test_s3_cluster/test.py` (`join_mode` `local` / `global`).
Categories failed	Documentation / user-visible consistency (see above).
Categories passed	Per-query external table forwarding; analyzer requirement for non-`allow` modes; no persistent join/external-table flag on `IStorageCluster` in reviewed code.
Assumptions / limits	Static review of the workspace; CI and full integration runs were not executed here.

arthurpassos

I am not familiar with this piece of code, I can't add any value within a short period of time. Given the release deadline, I'll trust the tests, AI review and approve it

alsugiliazova · 2026-03-27T13:14:42Z

Issue with FULL OUTER JOIN

Received exception from server (version 26.1.4):
Code: 264. DB::Exception: Received from localhost:9000. DB::Exception: Unexpected global join kind: FULL. (INCOMPATIBLE_TYPE_OF_JOIN)
(query: SELECT * FROM s3Cluster(replicated_cluster, 'http://minio:9000/warehouse/data1/data/**.parquet', '[masked]:Secret(name='minio_root_user')', '[masked]:Secret(name='minio_root_password')') AS t1
FULL OUTER JOIN merge_tree_table_86bbcd2c_29de_11f1_ae5b_de7b9eea348f AS t2 ON t1.float_col = t2.float_col ORDER BY tuple(*) SETTINGS object_storage_cluster_join_mode='global', object_storage_cluster='replicated_cluster_two_nodes' FORMAT Values)

ianton-ru added 4 commits February 25, 2026 11:55

object_storage_cluster_join_mode global

4a70d3b

Global gross join

972f1ee

Test for object_storage_cluster_join_mode='global'

b6bf151

Merge branch 'antalya-26.1' into feature/antalya-26.1/json_part2

38e89f4

ianton-ru added antalya antalya-26.1 labels Mar 13, 2026

chatgpt-codex-connector bot reviewed Mar 13, 2026

View reviewed changes

ianton-ru and others added 3 commits March 13, 2026 17:41

Check context existing before use

e15af07

Merge branch 'antalya-26.1' into feature/antalya-26.1/json_part2

7649388

Merge branch 'antalya-26.1' into feature/antalya-26.1/json_part2

cb5e474

chatgpt-codex-connector bot reviewed Mar 18, 2026

View reviewed changes

chatgpt-codex-connector bot reviewed Mar 23, 2026

View reviewed changes

Try to fix again

0cd90a8

ianton-ru force-pushed the feature/antalya-26.1/json_part2 branch from e3fe1a1 to 0cd90a8 Compare March 23, 2026 12:28

chatgpt-codex-connector bot reviewed Mar 23, 2026

View reviewed changes

ianton-ru and others added 2 commits March 23, 2026 16:37

Fix setting description

904d00e

Merge branch 'antalya-26.1' into feature/antalya-26.1/json_part2

d0de2cf

svb-alt added the antalya-26.1.6.20001 label Mar 25, 2026

svb-alt requested a review from arthurpassos March 26, 2026 12:25

arthurpassos approved these changes Mar 27, 2026

View reviewed changes

alsugiliazova mentioned this pull request Mar 27, 2026

Join Concurrency with object_storage_cluster_join_mode='global' #1591

Open

alsugiliazova added the verified-with-issue Verified by QA and issue(s) found. label Mar 27, 2026

alsugiliazova linked an issue Mar 27, 2026 that may be closed by this pull request

Join Concurrency with object_storage_cluster_join_mode='global' #1591

Open

		if (send_external_tables)
		external_tables = query_info.planner_context->getMutableQueryContext()->getExternalTables();

		bool prefer_local_join = getContext()->getSettingsRef()[Setting::parallel_replicas_prefer_local_join]
		&& getContext()->getSettingsRef()[Setting::object_storage_cluster_join_mode] != ObjectStorageClusterJoinMode::GLOBAL;

Conversation

ianton-ru commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

ianton-ru commented Mar 13, 2026

Uh oh!

github-actions bot commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Mar 18, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Mar 23, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Mar 23, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Mar 23, 2026

Audit: PR #1527 — Cluster Joins part 2 (global mode)

Confirmed defects

Low — Stale and misleading documentation for object_storage_cluster_join_mode

Notes (not classified as confirmed code defects)

Coverage summary

Uh oh!

arthurpassos left a comment

Choose a reason for hiding this comment

Uh oh!

alsugiliazova commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

ianton-ru commented Mar 13, 2026 •

edited

Loading

github-actions bot commented Mar 13, 2026 •

edited

Loading

Low — Stale and misleading documentation for `object_storage_cluster_join_mode`

alsugiliazova commented Mar 27, 2026 •

edited

Loading