Skip to content

Add support for Lookup Join on Multiple Fields #131559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 45 commits into from
Aug 13, 2025
Merged

Add support for Lookup Join on Multiple Fields #131559

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
b1ee80e
Lookup Join on Multiple Columns POC WIP
julian-elastic Jul 18, 2025
2c90817
Update docs/changelog/131559.yaml
julian-elastic Jul 18, 2025
4db37fd
Looking join on multiple fields WIP
julian-elastic Jul 21, 2025
ef894f3
Merge branch 'main' into lookupJoin
julian-elastic Jul 21, 2025
e367e8c
Fix more UTs
julian-elastic Jul 21, 2025
744479f
Bugfixes
julian-elastic Jul 21, 2025
c17c993
Bugfixes
julian-elastic Jul 21, 2025
d9462cc
Fix serialization error
julian-elastic Jul 21, 2025
a17fa88
Fix UT error
julian-elastic Jul 22, 2025
3fd48e4
Add more test datasets
julian-elastic Jul 22, 2025
64a07c7
Add UTs for join on 2,3,4 columns
julian-elastic Jul 22, 2025
ea171b1
Merge branch 'main' into lookupJoin
julian-elastic Jul 22, 2025
bae7007
Add handling for remote not supporting LOOKUP JOIN on multiple fields
julian-elastic Jul 23, 2025
6bd2937
Merge branch 'main' into lookupJoin
julian-elastic Jul 23, 2025
71adaa8
Change documentation
julian-elastic Jul 23, 2025
43aa7e1
Fix docs
julian-elastic Jul 24, 2025
7e0d8d7
Add more UTs
julian-elastic Jul 25, 2025
b6c615c
Merge branch 'main' into lookupJoin
julian-elastic Jul 25, 2025
5d9f68f
Address code review feedback
julian-elastic Jul 29, 2025
fc1c63b
Merge branch 'main' into lookupJoin
julian-elastic Jul 29, 2025
c179cea
Add Generative tests for Lookup Join On Multiple Columns
julian-elastic Jul 29, 2025
be2ce94
Merge branch 'main' into lookupJoin
julian-elastic Jul 29, 2025
59c16d9
Remove debugging code
julian-elastic Jul 29, 2025
dd52c02
Address a rare issue in Generative tests
julian-elastic Jul 30, 2025
8b2594b
Address docs issues
julian-elastic Jul 30, 2025
606c099
Merge branch 'main' into lookupJoin
julian-elastic Jul 30, 2025
e585342
Mode docs changes
julian-elastic Jul 30, 2025
72c3ad7
Merge branch 'main' into lookupJoin
julian-elastic Jul 30, 2025
f742160
Address code review feedback
julian-elastic Jul 30, 2025
ed6946b
Enhance LookupFromIndexIT
julian-elastic Jul 30, 2025
806933f
Fix failing UT
julian-elastic Jul 31, 2025
62956af
Merge branch 'main' into lookupJoin
julian-elastic Jul 31, 2025
326cb82
Address more code review comments
julian-elastic Jul 31, 2025
1dbe524
Address more code review comments, part 2
julian-elastic Jul 31, 2025
00b41ed
MatchConfig refactoring and add serialization test
julian-elastic Jul 31, 2025
a036aa6
bugfix
julian-elastic Jul 31, 2025
28d0c7c
Merge branch 'main' into lookupJoin
julian-elastic Jul 31, 2025
ce73957
Add HeapAttackIT cases with join on multiple fields
julian-elastic Aug 1, 2025
cfdb440
Merge branch 'main' into lookupJoin
julian-elastic Aug 1, 2025
96a6891
bugfix
julian-elastic Aug 1, 2025
d90799d
Update docs/changelog/131559.yaml
julian-elastic Aug 12, 2025
ec7f10a
Merge branch 'main' into lookupJoin
julian-elastic Aug 12, 2025
1cf34bc
Address code review comments
julian-elastic Aug 12, 2025
6acb8ef
Merge branch 'main' into lookupJoin
julian-elastic Aug 13, 2025
1102c1b
fix issue with docs
julian-elastic Aug 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ...i-clusters/src/javaRestTest/java/org/elasticsearch/xpack/esql/ccq/MultiClusterSpecIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ public MultiClusterSpecIT(
"SortBeforeAndAfterMultipleJoinAndMvExpand",
"LookupJoinAfterTopNAndRemoteEnrich",
"LookupJoinOnTwoFieldsAfterTop",
"LookupJoinOnTwoFieldsMultipleTimes",
// Lookup join after LIMIT is not supported in CCS yet
"LookupJoinAfterLimitAndRemoteEnrich"
);
Expand Down
3 changes: 1 addition & 2 deletions ...ava/org/elasticsearch/xpack/esql/qa/rest/generative/command/pipe/LookupJoinGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ public CommandDescription generate(
}
EsqlQueryGenerator.Column key = randomFrom(candidateKeys);
if (usedColumns.contains(key.name()) || usedColumns.contains(idxKey)) {
continue; // already used this column, skip
continue; // already used this column from the lookup index, or will discard the main index column by RENAME'ing below, skip
} else {
usedColumns.add(key.name());
usedColumns.add(idxKey);
Expand All @@ -74,7 +74,6 @@ public CommandDescription generate(
stringBuilder.append(", ");
}
}
stringBuilder.append("| LIMIT 10");
String cmdString = stringBuilder.toString();
return new CommandDescription(LOOKUP_JOIN, this, cmdString, Map.of());
}
Expand Down
6 changes: 4 additions & 2 deletions x-pack/plugin/esql/qa/testFixtures/src/main/resources/data/multi_column_joinable.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@ id_int,name_str,is_active_bool,ip_addr,extra1,extra2
9,Ivy,false,192.168.1.9,waldo,900
10,John,true,192.168.1.10,fred,1000
,Kate,false,192.168.1.11,plugh,1100
12,Liam,true,192.168.1.12,xyzzy,1200
[12],Liam,true,192.168.1.12,xyzzy,1200
13,Mia,false,192.168.1.13,thud,1300
14,Nina,true,192.168.1.14,foo2,1400
[14],Nina,true,192.168.1.14,foo2,1400
15,Oscar,false,192.168.1.15,bar2,1500
[17,18],Olivia,true,192.168.1.17,xyz,17000
[1,19,21],Sophia,true,192.168.1.21,zyx,21000
4 changes: 3 additions & 1 deletion x-pack/plugin/esql/qa/testFixtures/src/main/resources/data/multi_column_joinable_lookup.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,7 @@ id_int,name_str,is_active_bool,ip_addr,other1,other2
,Kate,false,192.168.1.11,mu,12000
12,Liam,true,192.168.1.12,nu,13000
13,Mia,false,192.168.1.13,xi,14000
14,Nina,true,192.168.1.14,omicron,15000
[14],Nina,true,192.168.1.14,omicron,15000
16,Paul,true,192.168.1.16,pi,16000
[17,18],Olivia,true,192.168.1.17,rho,17000
[1,19,20],Sophia,true,192.168.1.21,sigma,21000
99 changes: 99 additions & 0 deletions x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join.csv-spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4882,9 +4882,13 @@ FROM multi_column_joinable
| SORT id_int, name_str, extra1, other1, other2
;

warning:Line 2:3: evaluation of [LOOKUP JOIN multi_column_joinable_lookup ON id_int, name_str] failed, treating result as null. Only first 20 failures recorded.
warning:Line 2:3: java.lang.IllegalArgumentException: LOOKUP JOIN encountered multi-value

id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:integer
1 | Alice | foo | alpha | 1000
1 | Alice | foo | beta | 2000
[1, 19, 21] | Sophia | zyx | null | null
2 | Bob | bar | gamma | 3000
3 | Charlie | baz | delta | 4000
3 | Charlie | baz | epsilon | 5000
Expand All @@ -4900,9 +4904,50 @@ id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:int
13 | Mia | thud | xi | 14000
14 | Nina | foo2 | omicron | 15000
15 | Oscar | bar2 | null | null
[17, 18] | Olivia | xyz | null | null
null | Kate | plugh | null | null
;

lookupJoinOnTwoFieldsSelfJoin
required_capability: join_lookup_v12
required_capability: lookup_join_on_multiple_fields

FROM multi_column_joinable_lookup
| LOOKUP JOIN multi_column_joinable_lookup ON id_int, name_str
| KEEP id_int, name_str, other1, other2
| SORT id_int, name_str, other1, other2
;

warning:Line 2:3: evaluation of [LOOKUP JOIN multi_column_joinable_lookup ON id_int, name_str] failed, treating result as null. Only first 20 failures recorded.
warning:Line 2:3: java.lang.IllegalArgumentException: LOOKUP JOIN encountered multi-value

id_int:integer | name_str:keyword | other1:keyword | other2:integer
1 | Alice | alpha | 1000
1 | Alice | alpha | 1000
1 | Alice | beta | 2000
1 | Alice | beta | 2000
[1, 19, 20] | Sophia | null | null
2 | Bob | gamma | 3000
3 | Charlie | delta | 4000
3 | Charlie | delta | 4000
3 | Charlie | epsilon | 5000
3 | Charlie | epsilon | 5000
4 | David | zeta | 6000
5 | Eve | eta | 7000
5 | Eve | eta | 7000
5 | Eve | theta | 8000
5 | Eve | theta | 8000
6 | null | null | null
7 | Grace | kappa | 10000
8 | Hank | lambda | 11000
12 | Liam | nu | 13000
13 | Mia | xi | 14000
14 | Nina | omicron | 15000
16 | Paul | pi | 16000
[17, 18] | Olivia | null | null
null | Kate | null | null
;

lookupJoinOnThreeFields
required_capability: join_lookup_v12
required_capability: lookup_join_on_multiple_fields
Expand All @@ -4913,9 +4958,13 @@ FROM multi_column_joinable
| SORT id_int, name_str, extra1, other1, other2
;

warning:Line 2:3: evaluation of [LOOKUP JOIN multi_column_joinable_lookup ON id_int, name_str, is_active_bool] failed, treating result as null. Only first 20 failures recorded.
warning:Line 2:3: java.lang.IllegalArgumentException: LOOKUP JOIN encountered multi-value

id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:integer
1 | Alice | foo | alpha | 1000
1 | Alice | foo | beta | 2000
[1, 19, 21] | Sophia | zyx | null | null
2 | Bob | bar | gamma | 3000
3 | Charlie | baz | delta | 4000
4 | David | qux | zeta | 6000
Expand All @@ -4930,6 +4979,7 @@ id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:int
13 | Mia | thud | xi | 14000
14 | Nina | foo2 | omicron | 15000
15 | Oscar | bar2 | null | null
[17, 18] | Olivia | xyz | null | null
null | Kate | plugh | null | null
;

Expand All @@ -4944,8 +4994,12 @@ FROM multi_column_joinable
| SORT id_int, name_str, extra1, other1, other2
;

warning:Line 2:3: evaluation of [LOOKUP JOIN multi_column_joinable_lookup ON id_int, name_str, is_active_bool, ip_addr] failed, treating result as null. Only first 20 failures recorded.
warning:Line 2:3: java.lang.IllegalArgumentException: LOOKUP JOIN encountered multi-value

id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:integer
1 | Alice | foo | alpha | 1000
[1, 19, 21] | Sophia | zyx | null | null
2 | Bob | bar | null | null
3 | Charlie | baz | delta | 4000
4 | David | qux | zeta | 6000
Expand All @@ -4960,6 +5014,7 @@ id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:int
13 | Mia | thud | xi | 14000
14 | Nina | foo2 | omicron | 15000
15 | Oscar | bar2 | null | null
[17, 18] | Olivia | xyz | null | null
null | Kate | plugh | null | null
;

Expand All @@ -4976,6 +5031,7 @@ FROM multi_column_joinable

id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:integer | is_active_bool:boolean | ip_addr:ip
1 | Alice | foo | alpha | 1000 | true | 192.168.1.1
[1, 19, 20] | Sophia | zyx | sigma | 21000 | true | 192.168.1.21
3 | Charlie | baz | delta | 4000 | true | 192.168.1.3
4 | David | qux | zeta | 6000 | false | 192.168.1.4
5 | Eve | quux | eta | 7000 | true | 192.168.1.5
Expand All @@ -4985,6 +5041,7 @@ id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:int
12 | Liam | xyzzy | nu | 13000 | true | 192.168.1.12
13 | Mia | thud | xi | 14000 | false | 192.168.1.13
14 | Nina | foo2 | omicron | 15000 | true | 192.168.1.14
[17, 18] | Olivia | xyz | rho | 17000 | true | 192.168.1.17
null | Kate | plugh | mu | 12000 | false | 192.168.1.11
null | null | bar | null | null | false | 192.168.1.2
null | null | bar2 | null | null | false | 192.168.1.15
Expand All @@ -5004,6 +5061,11 @@ FROM multi_column_joinable
| SORT id_int, name_str, extra1, other1, other2
;

warning:Line 2:17: evaluation of [id_int + 5] failed, treating result as null. Only first 20 failures recorded.
warning:Line 2:17: java.lang.IllegalArgumentException: single-value function encountered multi-value
warning:Line 3:3: evaluation of [LOOKUP JOIN multi_column_joinable_lookup ON id_int, is_active_bool] failed, treating result as null. Only first 20 failures recorded.
warning:Line 3:3: java.lang.IllegalArgumentException: LOOKUP JOIN encountered multi-value

id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:integer
6 | null | foo | iota | 9000
7 | Grace | bar | kappa | 10000
Expand All @@ -5020,6 +5082,8 @@ id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:int
19 | null | foo2 | null | null
20 | null | bar2 | null | null
null | null | plugh | null | null
null | null | xyz | null | null
null | null | zyx | null | null
;


Expand All @@ -5037,6 +5101,9 @@ FROM multi_column_joinable
| SORT id_int, name_str, extra1, other1, other2
;

warning:Line 4:3: evaluation of [LOOKUP JOIN multi_column_joinable_lookup ON id_int, is_active_bool] failed, treating result as null. Only first 20 failures recorded.
warning:Line 4:3: java.lang.IllegalArgumentException: LOOKUP JOIN encountered multi-value

id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:integer
1 | Alice | foo | alpha | 1000
1 | Alice | foo | beta | 2000
Expand All @@ -5050,3 +5117,35 @@ id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:int
15 | null | bar2 | null | null
null | null | plugh | null | null
;


lookupJoinOnTwoFieldsMultipleTimes
required_capability: join_lookup_v12
required_capability: lookup_join_on_multiple_fields

FROM multi_column_joinable
| LOOKUP JOIN multi_column_joinable_lookup ON id_int, is_active_bool
| SORT extra1
| LIMIT 10
| LOOKUP JOIN multi_column_joinable_lookup ON name_str, is_active_bool
| KEEP id_int, name_str, extra1, other1, other2
| SORT id_int, name_str, extra1, other1, other2
;

warning:Line 2:3: evaluation of [LOOKUP JOIN multi_column_joinable_lookup ON id_int, is_active_bool] failed, treating result as null. Only first 20 failures recorded.
warning:Line 2:3: java.lang.IllegalArgumentException: LOOKUP JOIN encountered multi-value

id_int:integer | name_str:keyword | extra1:keyword | other1:keyword | other2:integer
1 | Alice | foo | alpha | 1000
1 | Alice | foo | alpha | 1000
1 | Alice | foo | beta | 2000
1 | Alice | foo | beta | 2000
2 | Bob | bar | gamma | 3000
3 | Charlie | baz | delta | 4000
7 | Grace | grault | kappa | 10000
8 | Hank | garply | lambda | 11000
14 | Nina | foo2 | omicron | 15000
null | null | bar2 | null | null
null | null | corge | null | null
null | null | fred | null | null
;
Loading
Loading