Skip to content

Commit 088a98f

Browse files
afoucretafoucret
authored
Merge branch 'main' into esql-inference-command-usage-limit
2 parents 0cd3208 + 0e7aca6 commit 088a98f

File tree

53 files changed

+1792
-500
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+1792
-500
lines changed

‎distribution/docker/src/docker/dockerfiles/cloud_ess_fips/Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
# Extract Elasticsearch artifact
2626
################################################################################
2727
28-
FROM docker.elastic.co/wolfi/chainguard-base-fips:latest@sha256:a02c67d96cd6ec1b50a055f1f5515e0987b643d001d436088cefc02e5b786bf9 AS builder
28+
FROM docker.elastic.co/wolfi/chainguard-base-fips:latest@sha256:f30b871373f23e31c9b083625d9e77075e6cde801227520b10b081b749b7f8c1 AS builder
2929
3030
# Install required packages to extract the Elasticsearch distribution
3131
RUN <%= retry.loop(package_manager, "export DEBIAN_FRONTEND=noninteractive && ${package_manager} update && ${package_manager} update && ${package_manager} add --no-cache curl") %>
@@ -104,7 +104,7 @@ WORKDIR /usr/share/elasticsearch/config
104104
# Add entrypoint
105105
################################################################################
106106

107-
FROM docker.elastic.co/wolfi/chainguard-base-fips:latest@sha256:a02c67d96cd6ec1b50a055f1f5515e0987b643d001d436088cefc02e5b786bf9
107+
FROM docker.elastic.co/wolfi/chainguard-base-fips:latest@sha256:f30b871373f23e31c9b083625d9e77075e6cde801227520b10b081b749b7f8c1
108108

109109
RUN <%= retry.loop(package_manager,
110110
"export DEBIAN_FRONTEND=noninteractive && \n" +

‎docs/changelog/137367.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 137367
2+
summary: GROUP BY ALL
3+
area: ES|QL
4+
type: enhancement
5+
issues: []

‎docs/changelog/138023.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 138023
2+
summary: Push down COUNT(*) BY DATE_TRUNC
3+
area: ES|QL
4+
type: feature
5+
issues: []

‎docs/changelog/138569.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 138569
2+
summary: Ignore abort-on-cleanup failure in S3 repo
3+
area: Snapshot/Restore
4+
type: bug
5+
issues: []

‎docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md‎

Lines changed: 6 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -13,28 +13,20 @@ Refer to [the high-level landing page](../../../../esql/esql-lookup-join.md) for
1313

1414
```esql
1515
FROM <source_index>
16-
| LOOKUP JOIN <lookup_index> ON <field_name>
17-
```
18-
19-
```esql
20-
FROM <source_index>
21-
| LOOKUP JOIN <lookup_index> ON <field_name1>, <field_name2>, <field_name3>
22-
```
23-
```esql
24-
FROM <source_index>
25-
| LOOKUP JOIN <lookup_index> ON <left_field1> >= <lookup_field1> AND <left_field2> == <lookup_field2>
16+
| LOOKUP JOIN <lookup_index> ON <join_condition>
2617
```
2718

2819
**Parameters**
2920

3021
`<lookup_index>`
3122
: The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster references are not supported. Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).
3223

33-
`<field_name>` or `<field_name1>, <field_name2>, <field_name3>` or `<left_field1> >= <lookup_field1> AND <left_field2> == <lookup_field2>`
34-
: The join condition. Can be one of the following:
24+
`<join_condition>`
25+
: Can be one of the following:
3526
* A single field name
36-
* A comma-separated list of field names {applies_to}`stack: ga 9.2`
37-
* An expression with one or more join conditions linked by `AND`. Each condition compares a field from the left index with a field from the lookup index using [binary operators](/reference/query-languages/esql/functions-operators/operators.md#esql-binary-operators) (`==`, `>=`, `<=`, `>`, `<`, `!=`). Each field name in the join condition must exist in only one of the indexes. Use RENAME to resolve naming conflicts. {applies_to}`stack: preview 9.2` {applies_to}`serverless: preview`
27+
* A comma-separated list of field names, for example `<field1>, <field2>, <field3>` {applies_to}`stack: ga 9.2`
28+
* An expression with one or more predicates linked by `AND`, for example `<left_field1> >= <lookup_field1> AND <left_field2> == <lookup_field2>`. Each predicate compares a field from the left index with a field from the lookup index using [binary operators](/reference/query-languages/esql/functions-operators/operators.md#esql-binary-operators) (`==`, `>=`, `<=`, `>`, `<`, `!=`). Each field name in the join condition must exist in only one of the indexes. Use RENAME to resolve naming conflicts. {applies_to}`stack: preview 9.2` {applies_to}`serverless: preview`
29+
* An expression that includes [full text functions](/reference/query-languages/esql/functions-operators/search-functions.md) and other Lucene-pushable functions, for example `MATCH(<lookup_field>, "search term") AND <left_field> == <lookup_field>`. These functions can be combined with binary operators and logical operators (`AND`, `OR`, `NOT`) to create complex join conditions. At least one condition that relates the lookup index fields to the left side of the join fields is still required. {applies_to}`stack: preview 9.3` {applies_to}`serverless: preview`
3830
: If using join on a single field or a field list, the fields used must exist in both your current query results and in the lookup index. If the fields contains multi-valued entries, those entries will not match anything (the added fields will contain `null` for those rows).
3931

4032

‎docs/reference/query-languages/esql/esql-lookup-join.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,10 @@ For example, you can use `LOOKUP JOIN` to:
3131
* You want to restrict users to use only specific lookup indices
3232
* You do not need to match using ranges or spatial relations
3333

34+
## Syntax reference
35+
36+
Refer to [`LOOKUP JOIN`](/reference/query-languages/esql/commands/lookup-join.md) for the detailed syntax reference.
37+
3438
## How the command works [esql-how-lookup-join-works]
3539

3640
The `LOOKUP JOIN` command adds fields from the lookup index as new columns to your results table based on matching values in the join field.
@@ -40,12 +44,14 @@ The command requires two parameters:
4044
* The join condition. Can be one of the following:
4145
* A single field name
4246
* A comma-separated list of field names {applies_to}`stack: ga 9.2`
43-
* An expression with one or more join conditions linked by `AND`. Each condition compares a field from the left index with a field from the lookup index using [binary operators](/reference/query-languages/esql/functions-operators/operators.md#esql-binary-operators) (`==`, `>=`, `<=`, `>`, `<`, `!=`). Each field name in the join condition must exist in only one of the indexes. Use RENAME to resolve naming conflicts. {applies_to}`stack: preview 9.2` {applies_to}`serverless: preview`
47+
* An expression with one or more join conditions linked by `AND`. {applies_to}`stack: preview 9.2` {applies_to}`serverless: preview`
48+
* An expression that includes [Full Text Functions](/reference/query-languages/esql/functions-operators/search-functions.md) and other Lucene pushable functions applied to fields from the lookup index {applies_to}`stack: preview 9.3` {applies_to}`serverless: preview`
4449

4550
```esql
4651
LOOKUP JOIN <lookup_index> ON <field_name> # Join on a single field
4752
LOOKUP JOIN <lookup_index> ON <field_name1>, <field_name2>, <field_name3> # Join on multiple fields
4853
LOOKUP JOIN <lookup_index> ON <left_field1> >= <lookup_field1> AND <left_field2> == <lookup_field2> # Join on expression
54+
LOOKUP JOIN <lookup_index> ON MATCH(lookup_field, "search term") AND <left_field> == <lookup_field> # Join with Full Text Functions
4955
```
5056

5157
:::{image} ../images/esql-lookup-join.png

‎modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3BlobContainer.java‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -608,7 +608,24 @@ private void executeMultipart(
608608
final String uploadId;
609609
try (AmazonS3Reference clientReference = s3BlobStore.clientReference()) {
610610
uploadId = clientReference.client().createMultipartUpload(createMultipartUpload(purpose, operation, blobName)).uploadId();
611-
cleanupOnFailureActions.add(() -> abortMultiPartUpload(purpose, uploadId, blobName));
611+
cleanupOnFailureActions.add(() -> {
612+
try {
613+
abortMultiPartUpload(purpose, uploadId, blobName);
614+
} catch (Exception e) {
615+
if (e instanceof SdkServiceException sdkServiceException
616+
&& sdkServiceException.statusCode() == RestStatus.NOT_FOUND.getStatus()) {
617+
// NOT_FOUND is what we wanted
618+
logger.atDebug()
619+
.withThrowable(e)
620+
.log("multipart upload of [{}] with ID [{}] not found on abort", blobName, uploadId);
621+
} else {
622+
// aborting the upload on failure is a best-effort cleanup step - if it fails then we must just move on
623+
logger.atWarn()
624+
.withThrowable(e)
625+
.log("failed to clean up multipart upload of [{}] with ID [{}] after earlier failure", blobName, uploadId);
626+
}
627+
}
628+
});
612629
}
613630
if (Strings.isEmpty(uploadId)) {
614631
throw new IOException("Failed to initialize multipart operation for " + blobName);

‎muted-tests.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -444,6 +444,9 @@ tests:
444444
- class: org.elasticsearch.xpack.esql.heap_attack.HeapAttackLookupJoinIT
445445
method: testLookupExplosionBigString
446446
issue: https://github.com/elastic/elasticsearch/issues/138510
447+
- class: org.elasticsearch.xpack.esql.optimizer.rules.physical.local.SubstituteRoundToTests
448+
method: testSubqueryWithCountStarAndDateTrunc {default}
449+
issue: https://github.com/elastic/elasticsearch/issues/138601
447450

448451
# Examples:
449452
#

‎server/src/test/java/org/elasticsearch/action/admin/cluster/stats/MappingStatsTests.java‎

Lines changed: 21 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ public class MappingStatsTests extends AbstractWireSerializingTestCase<MappingSt
9999

100100
private static final String SCRIPT_1 = scriptAsJSON("doc['field'] + doc.field + params._source.field");
101101
private static final String SCRIPT_2 = scriptAsJSON("doc['field']");
102-
private static final String SCRIPT_3 = scriptAsJSON("params._source.field + params._source.field + params._source.field");
102+
private static final String SCRIPT_3 = scriptAsJSON("params._source.field + params._source.field \n + params._source.field");
103103
private static final String SCRIPT_4 = scriptAsJSON("params._source.field");
104104

105105
public void testToXContent() {
@@ -114,8 +114,8 @@ public void testToXContent() {
114114
"mappings" : {
115115
"total_field_count" : 12,
116116
"total_deduplicated_field_count" : 6,
117-
"total_deduplicated_mapping_size" : "255b",
118-
"total_deduplicated_mapping_size_in_bytes" : 255,
117+
"total_deduplicated_mapping_size" : "260b",
118+
"total_deduplicated_mapping_size_in_bytes" : 260,
119119
"field_types" : [
120120
{
121121
"name" : "dense_vector",
@@ -159,10 +159,10 @@ public void testToXContent() {
159159
"lang" : [
160160
"painless"
161161
],
162-
"lines_max" : 1,
163-
"lines_total" : 4,
164-
"chars_max" : 66,
165-
"chars_total" : 172,
162+
"lines_max" : 2,
163+
"lines_total" : 6,
164+
"chars_max" : 68,
165+
"chars_total" : 176,
166166
"source_max" : 3,
167167
"source_total" : 8,
168168
"doc_max" : 0,
@@ -203,10 +203,10 @@ public void testToXContent() {
203203
"lang" : [
204204
"painless"
205205
],
206-
"lines_max" : 1,
207-
"lines_total" : 4,
208-
"chars_max" : 66,
209-
"chars_total" : 172,
206+
"lines_max" : 2,
207+
"lines_total" : 6,
208+
"chars_max" : 68,
209+
"chars_total" : 176,
210210
"source_max" : 3,
211211
"source_total" : 8,
212212
"doc_max" : 0,
@@ -246,8 +246,8 @@ public void testToXContentWithSomeSharedMappings() {
246246
"mappings" : {
247247
"total_field_count" : 18,
248248
"total_deduplicated_field_count" : 12,
249-
"total_deduplicated_mapping_size" : "513b",
250-
"total_deduplicated_mapping_size_in_bytes" : 513,
249+
"total_deduplicated_mapping_size" : "519b",
250+
"total_deduplicated_mapping_size_in_bytes" : 519,
251251
"field_types" : [
252252
{
253253
"name" : "dense_vector",
@@ -291,10 +291,10 @@ public void testToXContentWithSomeSharedMappings() {
291291
"lang" : [
292292
"painless"
293293
],
294-
"lines_max" : 1,
295-
"lines_total" : 6,
296-
"chars_max" : 66,
297-
"chars_total" : 258,
294+
"lines_max" : 2,
295+
"lines_total" : 9,
296+
"chars_max" : 68,
297+
"chars_total" : 264,
298298
"source_max" : 3,
299299
"source_total" : 12,
300300
"doc_max" : 0,
@@ -335,10 +335,10 @@ public void testToXContentWithSomeSharedMappings() {
335335
"lang" : [
336336
"painless"
337337
],
338-
"lines_max" : 1,
339-
"lines_total" : 6,
340-
"chars_max" : 66,
341-
"chars_total" : 258,
338+
"lines_max" : 2,
339+
"lines_total" : 9,
340+
"chars_max" : 68,
341+
"chars_total" : 264,
342342
"source_max" : 3,
343343
"source_total" : 12,
344344
"doc_max" : 0,

‎server/src/test/java/org/elasticsearch/index/codec/vectors/cluster/KMeansLocalTests.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@ public void testComputeNeighbours() throws IOException {
154154
vectors[i][j] = randomFloat();
155155
}
156156
}
157-
int clustersPerNeighbour = randomIntBetween(32, 128);
157+
int clustersPerNeighbour = randomIntBetween(64, 128);
158158
NeighborHood[] neighborHoodsGraph = NeighborHood.computeNeighborhoodsGraph(vectors, clustersPerNeighbour);
159159
NeighborHood[] neighborHoodsBruteForce = NeighborHood.computeNeighborhoodsBruteForce(vectors, clustersPerNeighbour);
160160
assertEquals(neighborHoodsGraph.length, neighborHoodsBruteForce.length);

0 commit comments

Comments
 (0)