Allow QueryRewriteContext to perform async actions on remote clusters #138124

Mikep86 · 2025-11-14T21:55:47Z

Adds methods to QueryRewriteContext for registering and executing remote cluster async actions as part of the query rewrite process.

This functionality will be used by the semantic and intercepted (match/sparse_vector/knn) queries to get remote cluster inference results when performing CCS with ccs_minimize_roundtrips=false. It will also be used by the simplified linear/rrf retrievers to handle CCS. See the semantic search CCS POC for more context.

Both of these use cases will be implemented in follow-up PRs.

elasticsearchmachine · 2025-11-14T21:56:11Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

server/src/main/java/org/elasticsearch/index/query/QueryRewriteContext.java

Mikep86 · 2025-11-14T21:58:11Z

@javanna

@jimczi asked me to add you to this PR to confirm that this will work with security

kderusso

Changes look reasonable to me, are there tests we can add here?

Mikep86 · 2025-11-17T15:49:48Z

@kderusso Tests were added in QueryRewriteContextMultiClustersIT, is there anything missing there?

kderusso · 2025-11-17T15:54:32Z

@Mikep86 Sorry for the confusion, were there any additional unit tests that could make sense.

Mikep86 · 2025-11-17T15:59:19Z

@kderusso Unit tests for QueryRewriteContext are tough because there's a ton of boilerplate you need to implement to create an instance for testing. And that's even before we consider all the mocking we would need to do to test async actions in a unit test. Due to all of this, I went straight to integration tests where we can use a real QueryRewriteContext instance and unmocked async actions.

kderusso · 2025-11-17T16:57:21Z

That's fair, I wanted to make sure we're thinking of it. Will defer to other reviewers for other concerns.

…curity enabled

…gement actions

Mikep86 · 2025-11-19T21:28:27Z

Moving this back into draft as I work on modifying the implementation and test to handle security

Mikep86 · 2025-11-19T22:32:03Z

@elasticmachine update branch

Mikep86 · 2025-11-20T19:17:21Z

@elasticmachine update branch

Mikep86 · 2025-11-20T19:42:04Z

Hey all, this is ready for review again. I made changes to AbstractMultiClustersTestCase and TestCluster to allow them to support security:

Use an internal client to perform internal cluster management operations. When security is enabled, this internal client needs to use an origin with sufficient privileges, which is enabled by calling internalClientOrigin.
Update AbstractMultiClustersTestCase to allow NodeConfigurationSource to be configured. SecuritySettingsSource is a convenient collection of all the settings required to enable security, and this change allows us to integrate it cleanly.

Mikep86 · 2025-11-20T19:46:33Z

...ternalClusterTest/java/org/elasticsearch/integration/QueryRewriteContextMultiClustersIT.java

+    private void deleteSecurityIndex(String clusterAlias) {
+        final Client client = new OriginSettingClient(client(clusterAlias), SECURITY_ORIGIN);
+
+        GetIndexRequest getIndexRequest = new GetIndexRequest(TEST_REQUEST_TIMEOUT);
+        getIndexRequest.indices(SECURITY_MAIN_ALIAS);
+        getIndexRequest.indicesOptions(IndicesOptions.lenientExpandOpen());
+        GetIndexResponse getIndexResponse = client.admin().indices().getIndex(getIndexRequest).actionGet(TEST_REQUEST_TIMEOUT);
+
+        if (getIndexResponse.getIndices().length > 0) {
+            DeleteIndexRequest deleteIndexRequest = new DeleteIndexRequest(getIndexResponse.getIndices());
+            assertAcked(client.admin().indices().delete(deleteIndexRequest).actionGet(TEST_REQUEST_TIMEOUT));
+        }
+    }


This is adapted from SecurityIntegTestCase: https://github.com/Mikep86/elasticsearch/blob/ef28a9da7a6d318c50b975ec4702314c6c4fb489/x-pack/plugin/security/src/test/java/org/elasticsearch/test/SecurityIntegTestCase.java#L413-L432

kderusso

Changes LGTM, as long as we have sufficient owner reviewers as well

server/src/main/java/org/elasticsearch/index/query/QueryRewriteContext.java

jimczi

LGTM

…elastic#138124)

Mikep86 added 14 commits November 12, 2025 10:15

Updated QueryRewriteContext to execute remote async actions

9ee1813

Resolve TODO

b6e329c

Added skeleton for QueryRewriteContextRemoteAsyncActionIT

034d1f5

Simplified test setup

a9534c1

Added test stubs

59ef48a

Implemented test query rewrite logic

c6b1e64

Renamed variables

1851f49

Fix warnings

77210cb

Implemented testCallRemoteAsyncAction

ffc99bb

Use search coordination thread pool

84e2e51

Implemented testInvalidClusterAlias

8d08360

Rename integration tests

9308455

Merge branch 'main' into query-rewrite_remote-async-actions

41c00af

Removed testRemoteClusterUnavailable

7590440

Mikep86 requested review from a team, javanna and jimczi November 14, 2025 21:55

Mikep86 added >non-issue :SearchOrg/Relevance Label for the Search (solution/org) Relevance team labels Nov 14, 2025

elasticsearchmachine added v9.3.0 Team:Search - Relevance The Search organization Search Relevance team labels Nov 14, 2025

Mikep86 commented Nov 14, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/index/query/QueryRewriteContext.java Outdated Show resolved Hide resolved

Mikep86 commented Nov 14, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/index/query/QueryRewriteContext.java Outdated Show resolved Hide resolved

kderusso reviewed Nov 17, 2025

View reviewed changes

Mikep86 added 2 commits November 19, 2025 16:21

Updated QueryRewriteContextMultiClustersIT to run with and without se…

6c57f3b

…curity enabled

Updated test cluster classes to use internal clients for cluster mana…

5116eea

…gement actions

Mikep86 marked this pull request as draft November 19, 2025 21:28

Mikep86 added 2 commits November 19, 2025 17:04

Fix license header

8c88ffb

Temporarily resolve jar hell

e6cc49d

elasticmachine and others added 9 commits November 19, 2025 23:32

Merge branch 'main' into query-rewrite_remote-async-actions

d859290

Make internal client origin configurable

ecda1fe

Cleanup security index

ab45505

Updated remote async actions to use thread context

19abecc

Make a copy of remote async actions map

a8c7661

Use a custom security settings source

1b9c788

Configure the client to use the test user when security is enabled

9596ecd

Updated test query to set origin

6afaec7

Added tests where origin is and is not set

a543197

Merge branch 'main' into query-rewrite_remote-async-actions

ef28a9d

Mikep86 marked this pull request as ready for review November 20, 2025 19:42

Mikep86 commented Nov 20, 2025

View reviewed changes

kderusso approved these changes Nov 21, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/index/query/QueryRewriteContext.java Outdated Show resolved Hide resolved

ioanatia approved these changes Nov 24, 2025

View reviewed changes

jimczi approved these changes Nov 24, 2025

View reviewed changes

Mikep86 added 3 commits November 24, 2025 15:20

Merge branch 'main' into query-rewrite_remote-async-actions

560475f

Temporarily disable testInvalidClusterAlias when security is enabled

dad30f5

Resolve TODO

4fa6794

Mikep86 merged commit 7e06534 into elastic:main Nov 24, 2025
34 checks passed

afoucret pushed a commit to afoucret/elasticsearch that referenced this pull request Nov 26, 2025

Allow QueryRewriteContext to perform async actions on remote clusters (…

8e0e358

…elastic#138124)

ncordon pushed a commit to ncordon/elasticsearch that referenced this pull request Nov 26, 2025

Allow QueryRewriteContext to perform async actions on remote clusters (…

a5ceefa

…elastic#138124)

Allow QueryRewriteContext to perform async actions on remote clusters #138124

Allow QueryRewriteContext to perform async actions on remote clusters #138124

Uh oh!

Conversation

Mikep86 commented Nov 14, 2025

Uh oh!

elasticsearchmachine commented Nov 14, 2025

Uh oh!

Uh oh!

Uh oh!

Mikep86 commented Nov 14, 2025

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Mikep86 commented Nov 17, 2025

Uh oh!

kderusso commented Nov 17, 2025

Uh oh!

Mikep86 commented Nov 17, 2025

Uh oh!

kderusso commented Nov 17, 2025

Uh oh!

Mikep86 commented Nov 19, 2025

Uh oh!

Mikep86 commented Nov 19, 2025

Uh oh!

Mikep86 commented Nov 20, 2025

Uh oh!

Mikep86 commented Nov 20, 2025

Uh oh!

Mikep86 Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jimczi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants