Remote Lookup Join implementation #129013

smalyshev · 2025-06-05T19:54:59Z

This patch enables using LOOKUP JOIN with cross-cluster queries. Example:

FROM logs-*, remote:logs-* | LOOKUP JOIN clients on ip | SORT timestamp | LIMIT 100

Currently some constructs are unsupported before LOOKUP JOIN when remote indices are involved:

SORT
LIMIT
STATS
ENRICH _coordinator:...
FORK

The LOOKUP JOIN with remote indices automatically assumes that the joins are executed remotely - i.e. LOOKUP JOIN with remote indices is forced to execute on the remote side of the query (inside FragmentExec) - this is the source of the most exclusions above.

elasticsearchmachine · 2025-06-30T18:08:18Z

Hi @smalyshev, I've created a changelog YAML for you.

elasticsearchmachine · 2025-07-01T00:01:47Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-07-01T00:01:47Z

Pinging @elastic/es-search-foundations (Team:Search Foundations)

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

alex-spies

All done now :) This is very nice!

I think this can nearly be merged, I would only make sure that we don't accidentally skip fwc tests as mentioned here. Other than that, my remarks are mostly on the minor side and also you already addressed a bunch of them :)

I mentioned some thoughts for follow-ups here. In addition, I saw many FIXMEs in the tests that I agree with. But that's for future work :)

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/join/LookupJoin.java

...ulti-clusters/src/javaRestTest/java/org/elasticsearch/xpack/esql/ccq/MultiClusterSpecIT.java

alex-spies · 2025-07-08T15:33:59Z

...ulti-clusters/src/javaRestTest/java/org/elasticsearch/xpack/esql/ccq/MultiClusterSpecIT.java

+                // If the query contains lookup indices, use only remotes to avoid duplication
+                onlyRemotes = true;


Err, doesn't this mean this suite doesn't at all test the case where a lookup happens both on remote and the local cluster?

This one doesn't because it wouldn't match the results - we'd have more rows than the local case. We have IT tests for that case. I haven't found any easy way to make these tests work in "both" case without substantially changing them.

...ulti-clusters/src/javaRestTest/java/org/elasticsearch/xpack/esql/ccq/MultiClusterSpecIT.java

alex-spies · 2025-07-08T16:25:00Z

...c/internalClusterTest/java/org/elasticsearch/xpack/esql/action/CrossClusterLookupJoinIT.java

+        ZonedDateTime nowUtc = ZonedDateTime.now(ZoneOffset.UTC);
+        ZonedDateTime nextMidnight = nowUtc.plusDays(1).withHour(0).withMinute(0).withSecond(0).withNano(0);
+        // If we're too close to midnight, we could create index with one day and query with another, and it'd fail.
+        assumeTrue("Skip if too close to midnight", Duration.between(nowUtc, nextMidnight).toMinutes() >= 5);


alex-spies · 2025-07-08T16:27:07Z

...c/internalClusterTest/java/org/elasticsearch/xpack/esql/action/CrossClusterLookupJoinIT.java

+            assertThat(columns, hasItems("lookup_key", "lookup_name", "lookup_tag", "v", "tag"));
+
+            List<List<Object>> values = getValuesList(resp);
+            assertThat(values, hasSize(10));


Yeah, I really don't know if a missing remote lookup index should lead to skipping the whole remote. That's a behavior we need to double check in the future.

...c/internalClusterTest/java/org/elasticsearch/xpack/esql/action/CrossClusterLookupJoinIT.java

alex-spies

Alright, this looks mighty good already. Let's

I'm not sure anymore if we already have a test that confirms that remote lookup joins remain disabled on non-SNAPSHOT builds. (This PR adds so many tests, I lost track. Nice test coverage, though!)

Could you please double check that we have such a test before merging? As a failsafe against accidentally enabling/releasing the feature prematurely.

smalyshev · 2025-07-09T15:28:26Z

I'm not sure anymore if we already have a test that confirms that remote lookup joins remain disabled on non-SNAPSHOT builds.

We don't have a specific test for that, how would I test that?

alex-spies · 2025-07-09T16:20:04Z

We don't have a specific test for that, how would I test that?

Hm, we could add an IT with assumeFalse(Build.current().isSnapshot()). This doesn't really require CCS infrastructure as we're supposed to fail during parsing, so we could just add this to org.elasticsearch.xpack.esql.qa.single_node.RestEsqlIT, maybe? It should fail when attempting to run FROM remote:idx | LOOKUP JOIN ... (or any other similar query).

Remote lookup join implementation This patch enables using LOOKUP JOIN with cross-cluster queries. Example: FROM logs-*, remote:logs-* | LOOKUP JOIN clients on ip | SORT timestamp | LIMIT 100

elasticsearchmachine added the v9.1.0 label Jun 5, 2025

smalyshev force-pushed the remote-lookup-join branch 3 times, most recently from cceb86d to bb3d64b Compare June 10, 2025 18:03

smalyshev force-pushed the remote-lookup-join branch from 29df38f to 59d98b2 Compare June 19, 2025 02:05

elasticsearchmachine added v9.2.0 and removed v9.1.0 labels Jun 26, 2025

github-actions bot deployed to docs-preview June 27, 2025 17:30 View deployment

smalyshev force-pushed the remote-lookup-join branch from cb90a8b to d8ec53d Compare June 27, 2025 17:43

smalyshev added >enhancement :Analytics/ES|QL AKA ESQL labels Jun 30, 2025

elastic deleted a comment from github-actions bot Jun 30, 2025

github-actions bot deployed to docs-preview June 30, 2025 18:08 View deployment

github-actions bot deployed to docs-preview June 30, 2025 18:11 View deployment

smalyshev added >feature and removed >enhancement labels Jun 30, 2025

github-actions bot deployed to docs-preview June 30, 2025 20:33 View deployment

github-actions bot deployed to docs-preview June 30, 2025 20:43 View deployment

github-actions bot deployed to docs-preview June 30, 2025 22:10 View deployment

smalyshev changed the title ~~WIP: Remote lookup join implementation~~ Remote lookup join implementation Jul 1, 2025

smalyshev marked this pull request as ready for review July 1, 2025 00:01

smalyshev added the :Search Foundations/CCS label Jul 1, 2025

smalyshev requested a review from alex-spies July 1, 2025 00:01

elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch labels Jul 1, 2025

github-actions bot deployed to docs-preview July 1, 2025 00:09 View deployment

idegtiarenko reviewed Jul 1, 2025

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java Outdated Show resolved Hide resolved

elasticsearchmachine and others added 3 commits July 7, 2025 21:20

[CI] Auto commit changes from spotless

d463df2

fix forbidden api

f58231e

Merge branch 'main' into remote-lookup-join

6597067

alex-spies reviewed Jul 8, 2025

View reviewed changes

smalyshev and others added 7 commits July 8, 2025 13:07

Move unsupported tests to Verifier

03a02fb

Improve lookup indices list

9858ef5

test fixes

a1b2bbf

Merge branch 'main' into remote-lookup-join

bd86864

use capability

9c90f92

moar tests

10b080f

Merge branch 'main' into remote-lookup-join

44a42ce

alex-spies approved these changes Jul 9, 2025

View reviewed changes

Merge branch 'main' into remote-lookup-join

d6d0f91

Clean up tests to ensure ENABLE_LOOKUP_JOIN_ON_REMOTE is respected

b11ab0c

smalyshev and others added 3 commits July 9, 2025 10:30

Add one more test

b170550

Merge branch 'main' into remote-lookup-join

44e1bf3

Merge branch 'main' into remote-lookup-join

288b7c4

smalyshev enabled auto-merge (squash) July 9, 2025 22:58

smalyshev added 3 commits July 9, 2025 18:51

Merge branch 'main' into remote-lookup-join

7d0fc12

Merge branch 'main' into remote-lookup-join

822b139

Merge branch 'main' into remote-lookup-join

8ecf11f

smalyshev merged commit e4155ea into elastic:main Jul 10, 2025
32 of 33 checks passed

alex-spies mentioned this pull request Jul 15, 2025

ESQL: Allow remote enrich after LOOKUP JOIN #131286

Closed

tylerperk added the release highlight label Jul 16, 2025

This was referenced Jul 18, 2025

ESQL: Disallow remote enrich after lu join #131426

Merged

ESQL: Refine PipelineBreaker interface #131507

Closed

		// If the query contains lookup indices, use only remotes to avoid duplication
		onlyRemotes = true;

Remote Lookup Join implementation #129013

Remote Lookup Join implementation #129013

Uh oh!

Conversation

smalyshev commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 30, 2025

Uh oh!

elasticsearchmachine commented Jul 1, 2025

Uh oh!

elasticsearchmachine commented Jul 1, 2025

Uh oh!

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-spies Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

smalyshev Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alex-spies Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

alex-spies Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

smalyshev commented Jul 9, 2025

Uh oh!

alex-spies commented Jul 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

smalyshev commented Jun 5, 2025 •

edited

Loading