Skip to content

Commit 8433712

Browse files
Tim-BrooksTim-Brooks
committed
Merge remote-tracking branch 'upstream/main' into structured_source
2 parents f7b771c + a995a12 commit 8433712

File tree

62 files changed

+1455
-436
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

62 files changed

+1455
-436
lines changed

‎.buildkite/pull-requests.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,19 @@
1515
"trigger_comment_regex": "(run\\W+elasticsearch-ci.+)|(^\\s*((buildkite|@elastic(search)?machine)\\s*)?test\\s+this(\\s+please)?)",
1616
"cancel_intermediate_builds": true,
1717
"cancel_intermediate_builds_on_comment": false
18+
},
19+
{
20+
"enabled": true,
21+
"pipeline_slug": "elasticsearch-performance-esbench-pr",
22+
"allow_org_users": true,
23+
"allowed_repo_permissions": [
24+
"admin",
25+
"write"
26+
],
27+
"set_commit_status": false,
28+
"build_on_commit": false,
29+
"build_on_comment": true,
30+
"trigger_comment_regex": "^(buildkite|@elastic(search)?machine) benchmark this with (?<benchmark>\\w+)( please)?$"
1831
}
1932
]
2033
}

‎docs/changelog/131559.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
pr: 131559
2+
summary: Add support for LOOKUP JOIN on multiple fields
3+
area: ES|QL
4+
type: enhancement
5+
issues: [ ]
6+
highlight:
7+
title: Add support for Lookup Join on Multiple Fields
8+
body: "Add support for Lookup Join on Multiple Fields e.g. FROM index1\n| LOOKUP\
9+
\ JOIN lookup_index on field1, field2"
10+
notable: true

‎docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,22 @@ FROM <source_index>
1717
| LOOKUP JOIN <lookup_index> ON <field_name>
1818
```
1919

20+
```esql
21+
FROM <source_index>
22+
| LOOKUP JOIN <lookup_index> ON <field_name1>, <field_name2>, <field_name3>
23+
```
24+
2025
**Parameters**
2126

2227
`<lookup_index>`
2328
: The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster references are not supported. Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).
2429

25-
`<field_name>`
26-
: The field to join on. This field must exist in both your current query results and in the lookup index. If the field contains multi-valued entries, those entries will not match anything (the added fields will contain `null` for those rows).
30+
`<field_name>` or `<field_name1>, <field_name2>, <field_name3>`
31+
: The field(s) to join on. Can be either:
32+
* A single field name
33+
* A comma-separated list of field names {applies_to}`stack: ga 9.2`
34+
: These fields must exist in both your current query results and in the lookup index. If the fields contains multi-valued entries, those entries will not match anything (the added fields will contain `null` for those rows).
35+
2736

2837
**Description**
2938

@@ -32,7 +41,7 @@ results table by finding documents in a lookup index that share the same
3241
join field value as your result rows.
3342

3443
For each row in your results table that matches a document in the lookup
35-
index based on the join field, all fields from the matching document are
44+
index based on the join fields, all fields from the matching document are
3645
added as new columns to that row.
3746

3847
If multiple documents in the lookup index match a single row in your

‎docs/reference/query-languages/esql/esql-lookup-join.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,11 +33,14 @@ For example, you can use `LOOKUP JOIN` to:
3333
The `LOOKUP JOIN` command adds fields from the lookup index as new columns to your results table based on matching values in the join field.
3434

3535
The command requires two parameters:
36-
- The name of the lookup index (which must have the `lookup` [`index.mode setting`](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting))
37-
- The name of the field to join on
38-
36+
* The name of the lookup index (which must have the `lookup` [`index.mode setting`](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting))
37+
* The field(s) to join on. Can be either:
38+
* A single field name
39+
* A comma-separated list of field names {applies_to}`stack: ga 9.2`
40+
3941
```esql
40-
LOOKUP JOIN <lookup_index> ON <field_name>
42+
LOOKUP JOIN <lookup_index> ON <field_name> # Join on a single field
43+
LOOKUP JOIN <lookup_index> ON <field_name1>, <field_name2>, <field_name3> # Join on multiple fields
4144
```
4245

4346
:::{image} ../images/esql-lookup-join.png
@@ -200,7 +203,7 @@ The following are the current limitations with `LOOKUP JOIN`:
200203
* Indices in [`lookup` mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting) are always single-sharded.
201204
* Cross cluster search is unsupported initially. Both source and lookup indices must be local.
202205
* Currently, only matching on equality is supported.
203-
* `LOOKUP JOIN` can only use a single match field and a single index. Wildcards are not supported.
206+
* In Stack versions `9.0-9.1`,`LOOKUP JOIN` can only use a single match field and a single index. Wildcards are not supported.
204207
* Aliases, datemath, and datastreams are supported, as long as the index pattern matches a single concrete index {applies_to}`stack: ga 9.1.0`.
205208
* The name of the match field in `LOOKUP JOIN lu_idx ON match_field` must match an existing field in the query. This may require `RENAME`s or `EVAL`s to achieve.
206209
* The query will circuit break if there are too many matching documents in the lookup index, or if the documents are too large. More precisely, `LOOKUP JOIN` works in batches of, normally, about 10,000 rows; a large amount of heap space is needed if the matching documents from the lookup index for a batch are multiple megabytes or larger. This is roughly the same as for `ENRICH`.

‎gradle/build.versions.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,5 +50,5 @@ wiremock = "com.github.tomakehurst:wiremock-jre8-standalone:2.23.2"
5050
xmlunit-core = "org.xmlunit:xmlunit-core:2.8.2"
5151

5252
[plugins]
53-
ospackage = { id = "com.netflix.nebula.ospackage-base", version = "11.11.2" }
53+
ospackage = { id = "com.netflix.nebula.ospackage-base", version = "12.1.0" }
5454
nmcp-aggregation = { id = "com.gradleup.nmcp.aggregation", version.ref="nmcp" }

‎gradle/verification-metadata.xml‎

Lines changed: 43 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -481,6 +481,11 @@
481481
<sha256 value="084197555590a53bb21b59508a3330559f536ddb448eafd1ec675f5462036fcf" origin="Generated by Gradle"/>
482482
</artifact>
483483
</component>
484+
<component group="com.github.luben" name="zstd-jni" version="1.5.6-3">
485+
<artifact name="zstd-jni-1.5.6-3.jar">
486+
<sha256 value="f72ede1b39258faf81277dc58de30c71cbae4253732558d2ce10b53d8b5763d5" origin="Generated by Gradle"/>
487+
</artifact>
488+
</component>
484489
<component group="com.github.spotbugs" name="spotbugs-annotations" version="4.0.2">
485490
<artifact name="spotbugs-annotations-4.0.2.jar">
486491
<sha256 value="3ef6c9f822b601aa151e10e123b49e5604243a4a99bcc47e4e1f9eea9781dc63" origin="Generated by Gradle"/>
@@ -1083,9 +1088,9 @@
10831088
<sha256 value="d694edd7bae3bc1a8e0dae2f5a22c479ff04d6b9bfcb0ab751a42f02e02d2100" origin="Generated by Gradle"/>
10841089
</artifact>
10851090
</component>
1086-
<component group="com.netflix.nebula" name="gradle-ospackage-plugin" version="11.11.2">
1087-
<artifact name="gradle-ospackage-plugin-11.11.2.jar">
1088-
<sha256 value="2cad0d0af05514e07b163d30d71c486ef1a205265ecc4821f8fb9a121f2cabe5" origin="Generated by Gradle"/>
1091+
<component group="com.netflix.nebula" name="gradle-ospackage-plugin" version="12.1.0">
1092+
<artifact name="gradle-ospackage-plugin-12.1.0.jar">
1093+
<sha256 value="825fbb7f3f52083eaae9bec50cee88da2a43462715b2e23afa27972ce803d092" origin="Generated by Gradle"/>
10891094
</artifact>
10901095
</component>
10911096
<component group="com.netflix.nebula" name="nebula-gradle-interop" version="2.0.0">
@@ -1533,6 +1538,11 @@
15331538
<sha256 value="b0a5159e926de8084ff066025142270443533656bc599b8bb31d14d11fd138a4" origin="Generated by Gradle"/>
15341539
</artifact>
15351540
</component>
1541+
<component group="io.airlift" name="aircompressor" version="0.27">
1542+
<artifact name="aircompressor-0.27.jar">
1543+
<sha256 value="fdbef3137a28f63bb0cb93487803080ede746a4ec3d421e36c6f0c305c35e5e4" origin="Generated by Gradle"/>
1544+
</artifact>
1545+
</component>
15361546
<component group="io.dropwizard.metrics" name="metrics-core" version="3.2.4">
15371547
<artifact name="metrics-core-3.2.4.jar">
15381548
<sha256 value="a083e3f3bdf4173d7065cdd4521964394a840d586bf8653676203d18354a6d2c" origin="Generated by Gradle"/>
@@ -2205,6 +2215,11 @@
22052215
<sha256 value="27bb5d40f37c3bb7205b4a0540247df057715e9f6cbbd97d626ab8b50318bb04" origin="Generated by Gradle"/>
22062216
</artifact>
22072217
</component>
2218+
<component group="org.apache.commons" name="commons-compress" version="1.26.2">
2219+
<artifact name="commons-compress-1.26.2.jar">
2220+
<sha256 value="9168a03141d8fc7eda21a2360d83cc0412bcbb1d6204d992bd48c2573cb3c6b8" origin="Generated by Gradle"/>
2221+
</artifact>
2222+
</component>
22082223
<component group="org.apache.commons" name="commons-compress" version="1.27.1">
22092224
<artifact name="commons-compress-1.27.1.jar">
22102225
<sha256 value="293d80f54b536b74095dcd7ea3cf0a29bbfc3402519281332495f4420d370d16" origin="Generated by Gradle"/>
@@ -3228,6 +3243,11 @@
32283243
<sha256 value="d0674469f8e6fbc866dbc5c955463d7629b7755ac7c87a235b9469d8257d6b8c" origin="Generated by Gradle"/>
32293244
</artifact>
32303245
</component>
3246+
<component group="org.apache.maven" name="maven-archiver" version="3.6.3">
3247+
<artifact name="maven-archiver-3.6.3.jar">
3248+
<sha256 value="83566129898c7a384bffaa118a76a40667d546948460b43eeb8563fa17635848" origin="Generated by Gradle"/>
3249+
</artifact>
3250+
</component>
32313251
<component group="org.apache.maven" name="maven-artifact" version="3.6.1">
32323252
<artifact name="maven-artifact-3.6.1.jar">
32333253
<sha256 value="797fa1bcf54d7216e863b8177c04507d09c708eec153626eca473054c0eb56f3" origin="Generated by Gradle"/>
@@ -3641,6 +3661,11 @@
36413661
<sha256 value="92654f493ecfec52082e76354f0ebf87648dc3d5cec2e3c3cdb947c016747a53" origin="Generated by Gradle"/>
36423662
</artifact>
36433663
</component>
3664+
<component group="org.codehaus.plexus" name="plexus-archiver" version="4.10.0">
3665+
<artifact name="plexus-archiver-4.10.0.jar">
3666+
<sha256 value="4c07814ff4a39199999ae82bba1e38aa4f25637467fcac6a66ed63a76535799a" origin="Generated by Gradle"/>
3667+
</artifact>
3668+
</component>
36443669
<component group="org.codehaus.plexus" name="plexus-archiver" version="4.2.3">
36453670
<artifact name="plexus-archiver-4.2.3.jar">
36463671
<sha256 value="fff8157bfb3d6f1f099c7da7de3e14bfd061ad4ab29599afa7bbf0271368d748" origin="Generated by Gradle"/>
@@ -3666,11 +3691,21 @@
36663691
<sha256 value="b3b5412ce17889103ea564bcdfcf9fb3dfa540344ffeac6b538a73c9d7182662" origin="Generated by Gradle"/>
36673692
</artifact>
36683693
</component>
3694+
<component group="org.codehaus.plexus" name="plexus-interpolation" version="1.27">
3695+
<artifact name="plexus-interpolation-1.27.jar">
3696+
<sha256 value="3fb4fb6143fdf964024c3cb738551524b9ea84e5c211cd660c559ad0703e5230" origin="Generated by Gradle"/>
3697+
</artifact>
3698+
</component>
36693699
<component group="org.codehaus.plexus" name="plexus-io" version="3.2.0">
36703700
<artifact name="plexus-io-3.2.0.jar">
36713701
<sha256 value="15cf8cbd9e014b7156482bbb48e515613158bdd9b4b908d21e6b900f7876f6ff" origin="Generated by Gradle"/>
36723702
</artifact>
36733703
</component>
3704+
<component group="org.codehaus.plexus" name="plexus-io" version="3.5.0">
3705+
<artifact name="plexus-io-3.5.0.jar">
3706+
<sha256 value="965ed28912cf1ae4c628112c4009e0c19819bc44ed5db8af54ee5eda21036a3e" origin="Generated by Gradle"/>
3707+
</artifact>
3708+
</component>
36743709
<component group="org.codehaus.plexus" name="plexus-utils" version="3.2.0">
36753710
<artifact name="plexus-utils-3.2.0.jar">
36763711
<sha256 value="0b91029df4c216b8824bd95361f52e260e86ccf93a2619fd88c8f15d23dcb30d" origin="Generated by Gradle"/>
@@ -4901,6 +4936,11 @@
49014936
<sha256 value="254cbb052f718f36ee2a75ae95bd2e0251b0513fc76d038b6dbbcf0942fcc11f" origin="Generated by Gradle"/>
49024937
</artifact>
49034938
</component>
4939+
<component group="org.vafer" name="jdeb" version="1.14">
4940+
<artifact name="jdeb-1.14.jar">
4941+
<sha256 value="2b999a40b769f6cca3fb381b8f1118fc344b88abe2f38ba1b2d53257f89f3602" origin="Generated by Gradle"/>
4942+
</artifact>
4943+
</component>
49044944
<component group="org.vafer" name="jdeb" version="1.8">
49054945
<artifact name="jdeb-1.8.jar">
49064946
<sha256 value="374fe02a09314da6ec8219e005598e760323a1abe4b715cc33ce3e2a9bcda932" origin="Generated by Gradle"/>

‎muted-tests.yml‎

Lines changed: 10 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -510,96 +510,33 @@ tests:
510510
- class: org.elasticsearch.gradle.internal.transport.TransportVersionManagementPluginFuncTest
511511
method: cannot change committed ids to a branch
512512
issue: https://github.com/elastic/elasticsearch/issues/132790
513-
- class: org.elasticsearch.xpack.esql.CsvTests
514-
method: test {csv-spec:spatial.ConvertFromStringParseError}
515-
issue: https://github.com/elastic/elasticsearch/issues/132805
516-
- class: org.elasticsearch.xpack.esql.CsvTests
517-
method: test {csv-spec:floats.ScalbWithHugeScaleFactor}
518-
issue: https://github.com/elastic/elasticsearch/issues/132806
519513
- class: org.elasticsearch.reservedstate.service.FileSettingsServiceIT
520514
method: testSettingsAppliedOnStart
521515
issue: https://github.com/elastic/elasticsearch/issues/131210
522516
- class: org.elasticsearch.index.mapper.vectors.SparseVectorFieldMapperTests
523517
method: testPruningScenarios
524518
issue: https://github.com/elastic/elasticsearch/issues/132810
525-
- class: org.elasticsearch.xpack.esql.CsvTests
526-
method: test {csv-spec:spatial_shapes.ConvertFromStringParseError}
527-
issue: https://github.com/elastic/elasticsearch/issues/132812
528-
- class: org.elasticsearch.xpack.esql.CsvTests
529-
method: test {csv-spec:string.MvStringNotEqualsLong}
530-
issue: https://github.com/elastic/elasticsearch/issues/132813
531-
- class: org.elasticsearch.xpack.esql.CsvTests
532-
method: test {csv-spec:spatial.ConvertCartesianFromStringParseError}
533-
issue: https://github.com/elastic/elasticsearch/issues/132814
534-
- class: org.elasticsearch.xpack.esql.CsvTests
535-
method: test {csv-spec:spatial_shapes.ConvertCartesianShapeFromStringParseError}
536-
issue: https://github.com/elastic/elasticsearch/issues/132815
537-
- class: org.elasticsearch.xpack.esql.CsvTests
538-
method: test {csv-spec:string.RepeatNegative}
539-
issue: https://github.com/elastic/elasticsearch/issues/132818
540-
- class: org.elasticsearch.xpack.esql.CsvTests
541-
method: test {csv-spec:stats.WeightedAvgWeightMvWarning}
542-
issue: https://github.com/elastic/elasticsearch/issues/132821
543-
- class: org.elasticsearch.xpack.esql.CsvTests
544-
method: test {csv-spec:ip.PushDownIPWithIn}
545-
issue: https://github.com/elastic/elasticsearch/issues/132822
546-
- class: org.elasticsearch.xpack.esql.CsvTests
547-
method: test {csv-spec:floats.ScalbWithHugeconstantFirstArgument}
548-
issue: https://github.com/elastic/elasticsearch/issues/132823
549-
- class: org.elasticsearch.xpack.esql.CsvTests
550-
method: test {csv-spec:boolean.MvSliceWarnings}
551-
issue: https://github.com/elastic/elasticsearch/issues/132824
552-
- class: org.elasticsearch.xpack.esql.CsvTests
553-
method: test {csv-spec:change_point.Values null column}
554-
issue: https://github.com/elastic/elasticsearch/issues/132825
555-
- class: org.elasticsearch.xpack.esql.CsvTests
556-
method: test {csv-spec:string.MvStringEqualsLongString}
557-
issue: https://github.com/elastic/elasticsearch/issues/132826
558-
- class: org.elasticsearch.xpack.esql.CsvTests
559-
method: test {csv-spec:stats.WeightedAvgFieldMvWarning}
560-
issue: https://github.com/elastic/elasticsearch/issues/132827
561519
- class: org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT
562520
method: test {p0=search/160_exists_query/Test exists query on mapped date field with no doc values}
563521
issue: https://github.com/elastic/elasticsearch/issues/132828
564522
- class: org.elasticsearch.test.rest.yaml.RcsCcsCommonYamlTestSuiteIT
565523
method: test {p0=search/160_exists_query/Test exists query on keyword field in empty index}
566524
issue: https://github.com/elastic/elasticsearch/issues/132829
567525
- class: org.elasticsearch.xpack.esql.CsvTests
568-
method: test {csv-spec:ip.CidrMatchFieldArg}
569-
issue: https://github.com/elastic/elasticsearch/issues/132834
570-
- class: org.elasticsearch.xpack.esql.CsvTests
571-
method: test {csv-spec:change_point.Null values}
572-
issue: https://github.com/elastic/elasticsearch/issues/132835
573-
- class: org.elasticsearch.xpack.esql.CsvTests
574-
method: test {csv-spec:ip.Conditional}
575-
issue: https://github.com/elastic/elasticsearch/issues/132836
576-
- class: org.elasticsearch.xpack.esql.CsvTests
577-
method: test {csv-spec:ip.CdirMatchOrsIPs}
578-
issue: https://github.com/elastic/elasticsearch/issues/132837
579-
- class: org.elasticsearch.xpack.esql.CsvTests
580-
method: test {csv-spec:string.SpaceNegative}
581-
issue: https://github.com/elastic/elasticsearch/issues/132838
582-
- class: org.elasticsearch.xpack.esql.CsvTests
583-
method: test {csv-spec:string.MvStringNotEqualsFound}
584-
issue: https://github.com/elastic/elasticsearch/issues/132839
585-
- class: org.elasticsearch.xpack.esql.CsvTests
586-
method: test {csv-spec:string.ConcatOfText}
587-
issue: https://github.com/elastic/elasticsearch/issues/132840
588-
- class: org.elasticsearch.xpack.esql.CsvTests
589-
method: test {csv-spec:floats.EqualToOrEqualToMultivalue}
590-
issue: https://github.com/elastic/elasticsearch/issues/132841
526+
method: test {csv-spec:floats.EqualToMultivalue}
527+
issue: https://github.com/elastic/elasticsearch/issues/132852
591528
- class: org.elasticsearch.xpack.esql.CsvTests
592-
method: test {csv-spec:mv_percentile.FromIndexPercentile}
593-
issue: https://github.com/elastic/elasticsearch/issues/132846
529+
method: test {csv-spec:string.MvStringEquals}
530+
issue: https://github.com/elastic/elasticsearch/issues/132856
594531
- class: org.elasticsearch.xpack.esql.CsvTests
595-
method: test {csv-spec:ip.CidrMatchFunctionArg}
596-
issue: https://github.com/elastic/elasticsearch/issues/132850
532+
method: test {csv-spec:string.LengthOfText}
533+
issue: https://github.com/elastic/elasticsearch/issues/132857
597534
- class: org.elasticsearch.xpack.esql.CsvTests
598-
method: test {csv-spec:conditional.CaseOnMv}
599-
issue: https://github.com/elastic/elasticsearch/issues/132851
535+
method: test {csv-spec:floats.InMultivalue}
536+
issue: https://github.com/elastic/elasticsearch/issues/132859
600537
- class: org.elasticsearch.xpack.esql.CsvTests
601-
method: test {csv-spec:floats.EqualToMultivalue}
602-
issue: https://github.com/elastic/elasticsearch/issues/132852
538+
method: test {csv-spec:ip.CdirMatchEqualsInsOrs}
539+
issue: https://github.com/elastic/elasticsearch/issues/132860
603540

604541
# Examples:
605542
#

‎server/src/main/java/org/elasticsearch/TransportVersions.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -363,7 +363,8 @@ static TransportVersion def(int id) {
363363
public static final TransportVersion INDEX_TEMPLATE_TRACKING_INFO = def(9_136_0_00);
364364
public static final TransportVersion EXTENDED_SNAPSHOT_STATS_IN_NODE_INFO = def(9_137_0_00);
365365
public static final TransportVersion SIMULATE_INGEST_MAPPING_MERGE_TYPE = def(9_138_0_00);
366-
public static final TransportVersion STRUCTURED_SOURCE = def(9_139_0_00);
366+
public static final TransportVersion ESQL_LOOKUP_JOIN_ON_MANY_FIELDS = def(9_139_0_00);
367+
public static final TransportVersion STRUCTURED_SOURCE = def(9_140_0_00);
367368

368369
/*
369370
* STOP! READ THIS FIRST! No, really,

‎server/src/main/java/org/elasticsearch/action/admin/indices/template/reservedstate/ReservedComposableIndexTemplateAction.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -182,7 +182,7 @@ public TransformState transform(ProjectId projectId, ComponentsAndComposables so
182182

183183
// 4. validate for v2 composable template overlaps
184184
for (var request : composables) {
185-
MetadataIndexTemplateService.v2TemplateOverlaps(project, request.name(), request.indexTemplate(), true);
185+
MetadataIndexTemplateService.v2TemplateOverlaps(project.templatesV2(), request.name(), request.indexTemplate(), true);
186186
}
187187

188188
Set<String> componentEntities = components.stream().map(r -> reservedComponentName(r.name())).collect(Collectors.toSet());

0 commit comments

Comments
 (0)