Add support for Lookup Join on Multiple Fields #131559

julian-elastic · 2025-07-18T21:02:08Z

Add support for Lookup Join on Multiple Fields

FROM index1
| LOOKUP JOIN lookup_index on field1, field2

Removed some checks to allow lookup join on multiple fields.
Added a new interface LookupEnrichQueryGenerator, that can be used to get total number of queries and queries by position. The rest of the methods from QueryGenerator are not needed by AbstractLookupService.

That allowed the creation of a new class ExpressionQueryList implements LookupEnrichQueryGenerator, which is responsible for creating the AND query for the different fields. We will likely need to enhance it in the future to support expressions that include OR and NOT as well.

TransportRequest is enhanced to now support List<MatchConfig> matchFields instead of String matchField. This is how we pass the match fields around now. If we are communicating with an cluster that does not support LookupOnMultipleFields and it is needed by the query we will fail the query. This can happen during rolling upgrade or CCS.

elasticsearchmachine · 2025-07-18T21:22:03Z

Hi @julian-elastic, I've created a changelog YAML for you.

github-actions · 2025-07-23T18:26:29Z

🔍 Preview links for changed docs

docs/reference/query-languages/esql/esql-lookup-join.md

elasticsearchmachine · 2025-07-23T20:22:47Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

alex-spies

Ok, I now got everything except some of the compute engine-only code. Looks good, but I think we can clean up the changed (transport) request a little more as it sends redundant data now.

alex-spies · 2025-07-31T08:34:39Z

docs/changelog/131559.yaml

We may want to declare this change as notable. I'm not sure what the bar is for that @leemthompo ?

If you and @tylerperk agree sounds like this passes the notable bar 😄 👍

@leemthompo How do I declare this change as notable? Can you point me to an example? Or you add it to some other release notes list after it is merged?

@alex-spies by notable you mean adding the release highlight label I guess?

@leemthompo I saw that we flag important release notes as notable, like here. Is this normally added via the release highlight label?

Both seem to make sense, so I'll go and mark this as release highlight and see what the bot does :)

flag important release notes as notable

TIL 😄

...esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/LookupFromIndexIT.java

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/enrich/LookupFromIndexOperatorTests.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/AbstractLookupService.java

...ck/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/LookupFromIndexService.java

nik9000

Looks good on my side. Left a few small things

...sql/compute/src/main/java/org/elasticsearch/compute/operator/lookup/ExpressionQueryList.java

x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join.csv-spec

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/LookupFromIndexOperator.java

...ck/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/LookupFromIndexService.java

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/LookupFromIndexOperator.java

leemthompo

Docs updates LGTM, thanks Julian

alex-spies

Super nice! Thanks for the iterations, I think this is a great PR @julian-elastic !

I have a final round of comments, of which I think the one about throwing the right exception should be addressed before merging but otherwise please proceed at your own discretion.

...heap-attack/src/javaRestTest/java/org/elasticsearch/xpack/esql/heap_attack/HeapAttackIT.java

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/LookupFromIndexOperator.java

...sql/compute/src/main/java/org/elasticsearch/compute/operator/lookup/ExpressionQueryList.java

elasticsearchmachine · 2025-08-12T09:36:47Z

Hi @julian-elastic, I've updated the changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

…-stats * upstream/main: (36 commits) ESQL: Fix async operator warnings not always sent when blocking (elastic#132744) Method not needed anymore (elastic#132912) [Test] Excercise shutdown more reliably in snapshot stress IT (elastic#132909) Update Gradle shadow plugin to 9.0.1 (elastic#132637) Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search/410_named_queries/named_queries_with_score} elastic#132906 Update docker.elastic.co/wolfi/chainguard-base-fips:latest Docker digest to fa6cb69 (elastic#132735) Remove unnecessary calls to fold() (elastic#131870) Use consistent terminology for transport version resources/references (elastic#132882) Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search.vectors/40_knn_search_cosine/kNN search only regular query} elastic#132890 Finalize release notes for v9.1.2 release (elastic#132745) Finalize release notes for v9.0.5 release (elastic#132718) Move inner records out of TransportVersionUtils (elastic#132872) Add support for Lookup Join on Multiple Fields (elastic#131559) Bootstrap PR-based benchmarks (elastic#132717) Refactor MetadataIndexTemplateService to use template maps instead of project metadata (elastic#132662) [Gradle] Update nebula ospackage plugin to 12.1.0 (elastic#132640) Mute org.elasticsearch.xpack.esql.CsvTests test {csv-spec:ip.CdirMatchEqualsInsOrs} elastic#132860 Mute org.elasticsearch.xpack.esql.CsvTests test {csv-spec:floats.InMultivalue} elastic#132859 Revert "Reuse prod code and reduce EsqlSession public surface" (elastic#132843) Mute org.elasticsearch.xpack.esql.CsvTests test {csv-spec:string.LengthOfText} elastic#132857 ...

* upstream/main: (278 commits) ESQL - dense vector support cosine normalization (elastic#132721) [ML] Add support for dimensions in google vertex ai request (elastic#132689) ESQL - Add byte element support for dense_vector data type (elastic#131863) ESQL: Fix async operator warnings not always sent when blocking (elastic#132744) Method not needed anymore (elastic#132912) [Test] Excercise shutdown more reliably in snapshot stress IT (elastic#132909) Update Gradle shadow plugin to 9.0.1 (elastic#132637) Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search/410_named_queries/named_queries_with_score} elastic#132906 Update docker.elastic.co/wolfi/chainguard-base-fips:latest Docker digest to fa6cb69 (elastic#132735) Remove unnecessary calls to fold() (elastic#131870) Use consistent terminology for transport version resources/references (elastic#132882) Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search.vectors/40_knn_search_cosine/kNN search only regular query} elastic#132890 Finalize release notes for v9.1.2 release (elastic#132745) Finalize release notes for v9.0.5 release (elastic#132718) Move inner records out of TransportVersionUtils (elastic#132872) Add support for Lookup Join on Multiple Fields (elastic#131559) Bootstrap PR-based benchmarks (elastic#132717) Refactor MetadataIndexTemplateService to use template maps instead of project metadata (elastic#132662) [Gradle] Update nebula ospackage plugin to 12.1.0 (elastic#132640) Mute org.elasticsearch.xpack.esql.CsvTests test {csv-spec:ip.CdirMatchEqualsInsOrs} elastic#132860 ...

Lookup Join on Multiple Columns POC WIP

b1ee80e

elasticsearchmachine added the v9.2.0 label Jul 18, 2025

julian-elastic added :Analytics/ES|QL AKA ESQL Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) >enhancement labels Jul 18, 2025

Update docs/changelog/131559.yaml

2c90817

julian-elastic added 3 commits July 21, 2025 08:28

Looking join on multiple fields WIP

4db37fd

Merge branch 'main' into lookupJoin

ef894f3

Fix more UTs

e367e8c

julian-elastic self-assigned this Jul 21, 2025

julian-elastic added 3 commits July 21, 2025 15:19

Bugfixes

744479f

Bugfixes

c17c993

Fix serialization error

d9462cc

julian-elastic force-pushed the lookupJoin branch from a66e89b to d9462cc Compare July 21, 2025 21:23

julian-elastic added 4 commits July 22, 2025 09:09

Fix UT error

a17fa88

Add more test datasets

3fd48e4

Add UTs for join on 2,3,4 columns

64a07c7

Merge branch 'main' into lookupJoin

ea171b1

julian-elastic force-pushed the lookupJoin branch from b3ac3af to ea171b1 Compare July 22, 2025 20:19

julian-elastic added 2 commits July 23, 2025 14:06

Add handling for remote not supporting LOOKUP JOIN on multiple fields

bae7007

Merge branch 'main' into lookupJoin

6bd2937

julian-elastic changed the title ~~Lookup Join on Multiple Columns POC WIP~~ Lookup Join on Multiple Fields Jul 23, 2025

Change documentation

71adaa8

julian-elastic marked this pull request as ready for review July 23, 2025 20:22

julian-elastic added 3 commits July 24, 2025 17:07

Fix docs

43aa7e1

Add more UTs

7e0d8d7

Merge branch 'main' into lookupJoin

b6c615c

julian-elastic added 2 commits July 30, 2025 20:29

Fix failing UT

806933f

Merge branch 'main' into lookupJoin

62956af

julian-elastic requested a review from alex-spies July 31, 2025 01:44

alex-spies reviewed Jul 31, 2025

View reviewed changes

nik9000 reviewed Jul 31, 2025

View reviewed changes

Address more code review comments

326cb82

leemthompo reviewed Jul 31, 2025

View reviewed changes

julian-elastic marked this pull request as draft July 31, 2025 16:34

julian-elastic added 7 commits July 31, 2025 12:36

Address more code review comments, part 2

1dbe524

MatchConfig refactoring and add serialization test

00b41ed

bugfix

a036aa6

Merge branch 'main' into lookupJoin

28d0c7c

Add HeapAttackIT cases with join on multiple fields

ce73957

Merge branch 'main' into lookupJoin

cfdb440

bugfix

96a6891

julian-elastic marked this pull request as ready for review August 1, 2025 14:22

nik9000 approved these changes Aug 1, 2025

View reviewed changes

julian-elastic requested a review from alex-spies August 2, 2025 00:28

alex-spies approved these changes Aug 7, 2025

View reviewed changes

alex-spies added the release highlight label Aug 12, 2025

Update docs/changelog/131559.yaml

d90799d

julian-elastic added 4 commits August 12, 2025 09:27

Merge branch 'main' into lookupJoin

ec7f10a

Address code review comments

1cf34bc

Merge branch 'main' into lookupJoin

6acb8ef

fix issue with docs

1102c1b

julian-elastic merged commit a995a12 into elastic:main Aug 13, 2025
33 checks passed

julian-elastic deleted the lookupJoin branch August 13, 2025 18:17

Add support for Lookup Join on Multiple Fields #131559

Add support for Lookup Join on Multiple Fields #131559

Uh oh!

Conversation

julian-elastic commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 18, 2025

Uh oh!

github-actions bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

elasticsearchmachine commented Jul 23, 2025

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

alex-spies Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

leemthompo Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julian-elastic Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

leemthompo Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

alex-spies Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

leemthompo Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nik9000 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leemthompo left a comment

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

julian-elastic commented Jul 18, 2025 •

edited

Loading

github-actions bot commented Jul 23, 2025 •

edited

Loading

leemthompo Jul 31, 2025 •

edited

Loading

leemthompo Aug 12, 2025 •

edited

Loading