Antalya 26.1: Remote initiator improvements by ianton-ru · Pull Request #1577 · Altinity/ClickHouse

ianton-ru · 2026-03-25T12:57:46Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Different improvement for remote initiator

Documentation entry for user-facing changes

With remote initiator feature queries like

SELECT * FROM iceberg(...) SETTINGS object_storage_cluster='swarm', object_storage_remote_initiator=1

rewrites as

SELECT * FROM remote('remote_host', icebergCluster('swarm', ...)

'remote_host' is a random host from 'swarm' cluster
See #756

Current PR introduces the next improvements:

Partially solved object_storage_remote_initiator auth works incorrectly #1570 - uses username and password if access to cluster requires it. Throws exception if cluster uses common secret, this should be solved in future PRs.
Solved object_storage_remote_initiator with different cluster name #1571 - new setting object_storage_remote_initiator_cluster allows to choose remote_host from different cluster, not only from swarm
remote query did not work with additional setting inside function, like remote('remote_host', iceberg(..., SETTINGS iceberg_metadata_file_path='path/to/metadata.json')). Now must work correctly.
In query remote('remote_host', icebergCluster('remote_cluster', ...)) cluster remote_cluster can be defined only on remote_host and unknown on current initial host. Removed cluster check on early stage, this allows to execute such queries.

CI/CD Options

Exclude tests:

Regression jobs to run:

github-actions · 2026-03-25T12:59:43Z

Workflow [PR], commit [259fb32]

ianton-ru · 2026-03-25T12:59:54Z

@codex review

ianton-ru · 2026-03-25T13:08:17Z

Audit: PR #1577 — Antalya 26.1: Remote initiator improvements

Source: Altinity/ClickHouse#1577
Base: antalya-26.1
Reviewed revision: branch including commit d7c4beebbdb (Remove unused header) and prior feature commits.

Confirmed defects

None. No confirmed defects in reviewed scope.

(Earlier revision briefly introduced an unused #include <Common/logger_useful.h> in QueryAnalyzer.cpp; commit d7c4beebbdb removes it.)

Coverage summary

Area	Notes
Scope reviewed	`FunctionNode` SETTINGS handling (`QueryTreeBuilder`, `FunctionNode` hash / clone / equal / `toASTImpl`), `QueryAnalyzer::resolveTableFunction` → `TableFunctionNode` bridge, `StorageDistributed::buildQueryTreeDistributed`, `IStorageCluster::{read, convertToRemote}` (initiator cluster, secret guard, URI decode, user/password `remote`), `ITableFunctionCluster` cluster-presence check removal, `Settings` / `SettingsChangesHistory`, integration tests under `tests/integration/test_s3_cluster/`.
Categories failed	None.
Categories passed	SETTINGS propagation for nested table functions; initiator cluster selection (`object_storage_remote_initiator_cluster`); host decode and auth branches; fail-closed on cluster secret (`NOT_IMPLEMENTED`); deferred cluster resolution for remote-only definitions; recursion guard via `object_storage_remote_initiator = false` on copied context; `SettingsChangesHistory` updates; integration scenarios (secret error, split initiator vs object-storage cluster).
Assumptions / limits	Pre-existing `static pcg64` RNG in `convertToRemote` may be a concurrency concern under parallel queries; not introduced by this PR. `Poco::URI::decode` on full `Address::toString()` not exhaustively validated for every exotic address encoding; aligned with `escapeForFileName` / dotted-host tests.

Audit metadata

Method: Static review per workspace audit-review standard (call graph, transitions, fault categories, concurrency and C++ hazard pass).
Output rule: Confirmed defects only; hypotheticals folded into assumptions/limits.

AI-generated audit note.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a0d1972000

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-25T13:16:43Z

src/Storages/IStorageCluster.cpp

+        auto remote_initiator_cluster_name = settings[Setting::object_storage_remote_initiator_cluster].value;
+        if (remote_initiator_cluster_name.empty())
+            remote_initiator_cluster_name = cluster_name_from_settings;
+        auto remote_initiator_cluster = getClusterImpl(context, remote_initiator_cluster_name);
+        auto storage_and_context = convertToRemote(remote_initiator_cluster, context, remote_initiator_cluster_name, query_to_send);


Strip initiator-only cluster setting before forwarding query

When object_storage_remote_initiator_cluster is set, the query forwarded via remote(...) still carries that setting even though convertToRemote only clears object_storage_remote_initiator. This makes remote execution depend on remote nodes understanding a setting that is only needed on the initiator; in mixed-version/rolling-upgrade clusters, older remote hosts can fail with unknown-setting errors before execution. The forwarded AST settings should drop object_storage_remote_initiator_cluster together with object_storage_remote_initiator.

Useful? React with 👍 / 👎.

@ianton-ru does it make sense? it looks like it does

object_storage_remote_initiator_cluster does nothing without ``object_storage_remote_initiator`. Make sense to remove just for less garbage in sub-query.

ianton-ru · 2026-03-26T11:29:26Z

tests/queries/0_stateless/01625_constraints_index_append.reference

      Prewhere filter
      Prewhere filter column: less(multiply(2, b), 100)
-  Filter column: and(indexHint(greater(plus(i, 40), 0)), equals(a, 0)) (removed)
+  Filter column: and(equals(a, 0), indexHint(greater(plus(i, 40), 0))) (removed)


Argument order depends on hash, hash was changes (see FunctionNode::updateTreeHashImpl)

arthurpassos · 2026-03-26T18:50:17Z

src/Storages/IStorageCluster.cpp

    if (settings[Setting::object_storage_remote_initiator])
    {
-        auto storage_and_context = convertToRemote(cluster, context, cluster_name_from_settings, query_to_send);
+        auto remote_initiator_cluster_name = settings[Setting::object_storage_remote_initiator_cluster].value;


Please add a comment explaining what this code block does. It took me a while to understand it by just reading the code.

I suggest something like:

/// In case the current node is not supposed to initiate the clustered query /// Sends this query to a remote initiator using the `remote` table function if (settings[Setting::object_storage_remote_initiator]) { /// Re-writes queries in the form of: /// Input: SELECT * FROM iceberg(...) SETTINGS object_storage_cluster='swarm', object_storage_remote_initiator=1 /// Output: SELECT * FROM remote('remote_host', icebergCluster('swarm', ...) /// Where `remote_host` is a random host from the cluster which will execute the query /// This means the initiator node belongs to the same cluster that will execute the query /// In case remote_initiator_cluster_name is set, the initiator might be set to a different cluster }

arthurpassos · 2026-03-26T18:50:50Z

src/Storages/IStorageCluster.cpp

    if (shard_addresses.size() != 1)
        throw Exception(ErrorCodes::LOGICAL_ERROR, "Size of shard {} in cluster {} is not equal 1", shard_num, cluster_name_from_settings);
-    auto host_name = shard_addresses[0].toString();
+    std::string host_name;


Address here in encoded format foo%2Ebar instead of foo.bar. This wasn't catched in tests before I add a host with dot in name.

arthurpassos · 2026-03-26T18:52:16Z

src/Storages/IStorageCluster.cpp

-    auto remote_query = makeASTFunction(remote_function_name, make_intrusive<ASTLiteral>(host_name), table_expression->table_function);
+    boost::intrusive_ptr<ASTFunction> remote_query;
+
+    if (shard_addresses[0].user_specified)


A comment please

arthurpassos · 2026-03-26T18:54:24Z

src/TableFunctions/ITableFunctionCluster.h

-            throw Exception(ErrorCodes::CLUSTER_DOESNT_EXIST, "Requested cluster '{}' not found", cluster_name);
+        /// Remove check cluster existing here
+        /// In query like
+        /// remote('remote_host', xxxCluster('remote_cluster', ...))


What if the query is not remote? Can't we check for that?

/// If cluster not exists, query falls later

Where and with which exception? It would be good to avoid any network calls before failing

Cluster name is not a network node name, it's an internal ClickHouse name. Query falls later when tries to get hosts from cluster. Network calls can't be made without hosts.

But it's hard to understand here is cluster function inside 'remote' or not.

arthurpassos · 2026-03-26T18:58:51Z

src/Storages/IStorageCluster.cpp

+        auto remote_initiator_cluster_name = settings[Setting::object_storage_remote_initiator_cluster].value;
+        if (remote_initiator_cluster_name.empty())
+            remote_initiator_cluster_name = cluster_name_from_settings;
+        auto remote_initiator_cluster = getClusterImpl(context, remote_initiator_cluster_name);
+        auto storage_and_context = convertToRemote(remote_initiator_cluster, context, remote_initiator_cluster_name, query_to_send);


@ianton-ru does it make sense? it looks like it does

arthurpassos · 2026-03-26T19:14:00Z

The changes look ok, but I think it needs more documentation. I also wonder if we can keep the 'cluster exists' check by wrapping that check in a `if (!is_remote0'

…r_improvements

arthurpassos

LGTM

ianton-ru added 5 commits March 24, 2026 09:49

User/password auth for object_storage_remote_initiator

000a737

Keep function settings in remote call

d4b850d

object_storage_remote_initiator_cluster setting

0c95253

Fix remote query with clustre, unknown on initial node

8dab965

Fix FunctionNode::toASTImpl

a0d1972

ianton-ru added antalya antalya-26.1 labels Mar 25, 2026

Remove unused header

d7c4bee

chatgpt-codex-connector bot reviewed Mar 25, 2026

View reviewed changes

svb-alt added the antalya-26.1.6.20001 label Mar 25, 2026

Fix test

df2595a

ianton-ru commented Mar 26, 2026

View reviewed changes

svb-alt requested a review from arthurpassos March 26, 2026 12:33

arthurpassos reviewed Mar 26, 2026

View reviewed changes

ianton-ru and others added 2 commits March 27, 2026 15:51

Add comments

d8b6b82

Merge branch 'antalya-26.1' into feature/antalya-26.1/remote_initiato…

259fb32

…r_improvements

arthurpassos approved these changes Mar 27, 2026

View reviewed changes

Conversation

ianton-ru commented Mar 25, 2026

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianton-ru commented Mar 25, 2026

Uh oh!

ianton-ru commented Mar 25, 2026

Audit: PR #1577 — Antalya 26.1: Remote initiator improvements

Confirmed defects

Coverage summary

Audit metadata

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arthurpassos commented Mar 26, 2026

Uh oh!

arthurpassos left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Mar 25, 2026 •

edited

Loading