Merge branch 'main' into fix-129148-take2

albertzaharovits · albertzaharovits · commit 88c9d84a7578 · 2025-06-25T16:32:53.000+03:00
diff --git a/docs/extend/creating-classic-plugins.md b/docs/extend/creating-classic-plugins.md
@@ -136,13 +136,12 @@ org.example.module: # or 'ALL-UNNAMED' if the plugin is non-modular
 Allows code to access the filesystem, to read or write paths as specified by the entitlement's fields. The filesystem of the OS hosting {{es}} may contain sensitive files, for example credentials. Some files are meant to be always accessible to {{es}}, but plugins can not access them directly: {{es}} enforces that certain files can only be read by its core code, while some other files can not be read or written at all. A plugin is always granted `read` access to the {{es}} config directory and `read_write` access to the temp directory; if the plugin requires to read, write or access additional files or directories, it must specify them via this entitlement.
 
 It is possible to specify 3 different types of file entitlement:
-  - `path` to specify an absolute path
-  - `relative_path` to specify a relative path. The path will be resolved via the `relative_to` field, which is used to qualify the relative path. It can be a specific {{es}} directory (`config` or `data`), or to the user home directory (`home`) (the home of the user running {{es}})
-  - `relative_path` to specify a path resolved via the `relative_to` field, which can have the following values:
-    - `config`: the {{es}} [config directory](https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html#config-files-location)
-    - `data`: the {{es}} [data directory](https://www.elastic.co/guide/en/elasticsearch/reference/current/path-settings-overview.html)
-    - `home`: the home directory of the user running {{es}}
-  - `path_setting` to specify a path defined via an {{es}} setting. The path can be absolute or relative; in the latter case, the path will be resolved using the `basedir_if_relative` path (which can assume the same values as `relative_to`)
+1. `path` to specify an absolute path
+2. `relative_path` to specify a relative path. Use the `relative_to` field to qualify the relative path. `relative_to` accepts the following options:
+   - `config`: the {{es}} [config directory](https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html#config-files-location)
+   - `data`: the {{es}} [data directory](https://www.elastic.co/guide/en/elasticsearch/reference/current/path-settings-overview.html)
+   - `home`: the home directory of the user running {{es}}
+3. `path_setting` to specify a path defined via an {{es}} setting. The path can be absolute or relative; in the latter case, the path will be resolved using the `basedir_if_relative` path (which can assume the same values as `relative_to`)
 
 Each of the 3 types has some additional fields:
 - `mode` (required): can be either `read` or `read_write`
diff --git a/docs/reference/query-languages/esql/esql-metadata-fields.md b/docs/reference/query-languages/esql/esql-metadata-fields.md
@@ -6,24 +6,40 @@ mapped_pages:
 
 # {{esql}} metadata fields [esql-metadata-fields]
 
+{{esql}} can access [metadata fields](/reference/elasticsearch/mapping-reference/document-metadata-fields.md).
 
-{{esql}} can access [metadata fields](/reference/elasticsearch/mapping-reference/document-metadata-fields.md). The currently supported ones are:
-
-* [`_index`](/reference/elasticsearch/mapping-reference/mapping-index-field.md): the index to which the document belongs. The field is of the type [keyword](/reference/elasticsearch/mapping-reference/keyword.md).
-* [`_id`](/reference/elasticsearch/mapping-reference/mapping-id-field.md): the source document’s ID. The field is of the type [keyword](/reference/elasticsearch/mapping-reference/keyword.md).
-* `_version`: the source document’s version. The field is of the type [long](/reference/elasticsearch/mapping-reference/number.md).
-* [`_ignored`](/reference/elasticsearch/mapping-reference/mapping-ignored-field.md): the ignored source document fields. The field is of the type [keyword](/reference/elasticsearch/mapping-reference/keyword.md).
-* `_score`: when enabled, the final score assigned to each row matching an ES|QL query. Scoring will be updated when using [full text search functions](/reference/query-languages/esql/functions-operators/search-functions.md).
-
-To enable the access to these fields, the [`FROM`](/reference/query-languages/esql/commands/source-commands.md#esql-from) source command needs to be provided with a dedicated directive:
+To access these fields, use the `METADATA` directive with the [`FROM`](/reference/query-languages/esql/commands/source-commands.md#esql-from) source command. For example:
 
 ```esql
 FROM index METADATA _index, _id
 ```
 
-Metadata fields are only available if the source of the data is an index. Consequently, `FROM` is the only source commands that supports the `METADATA` directive.
+## Available metadata fields
+
+The following metadata fields are available in {{esql}}:
+
+| Metadata field | Type | Description |
+|---------------|------|-------------|
+| [`_id`](/reference/elasticsearch/mapping-reference/mapping-id-field.md) | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | Unique document ID. |
+| [`_ignored`](/reference/elasticsearch/mapping-reference/mapping-ignored-field.md) | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | Names every field in a document that was ignored when the document was indexed. |
+| [`_index`](/reference/elasticsearch/mapping-reference/mapping-index-field.md) | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | Index name. |
+| `_index_mode` | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | [Index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting). For example: `standard`, `lookup`, or `logsdb`. |
+| `_score` | [`float`](/reference/elasticsearch/mapping-reference/number.md) | Query relevance score (when enabled). Scores are updated when using [full text search functions](/reference/query-languages/esql/functions-operators/search-functions.md). |
+| [`_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md) | Special `_source` type | Original JSON document body passed at index time (or a reconstructed version if [synthetic `_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source) is enabled). |
+| `_version` | [`long`](/reference/elasticsearch/mapping-reference/number.md) | Document version number |
+
+## Usage and limitations
+
+- Metadata fields are only available when the data source is an index
+- The `_source` type is not supported by functions
+- Only the `FROM` command supports the `METADATA` directive
+- Once enabled, metadata fields work like regular index fields
+
+## Examples
 
-Once enabled, these fields will be available to subsequent processing commands, just like other index fields:
+### Basic metadata usage
+
+Once enabled, metadata fields are available to subsequent processing commands, just like other index fields:
 
 ```esql
 FROM ul_logs, apps METADATA _index, _version
@@ -40,6 +56,8 @@ FROM ul_logs, apps METADATA _index, _version
 | 14 | apps | 1 | apps_14 |
 | 14 | ul_logs | 1 | ul_logs_14 |
 
+### Metadata fields and aggregations
+
 Similar to index fields, once an aggregation is performed, a metadata field will no longer be accessible to subsequent commands, unless used as a grouping field:
 
 ```esql
@@ -51,3 +69,15 @@ FROM employees METADATA _index, _id
 | --- | --- |
 | 10100 | employees |
 
+### Sort results by search score
+
+```esql
+FROM products METADATA _score
+| WHERE MATCH(description, "wireless headphones")
+| SORT _score DESC
+| KEEP name, description, _score
+```
+
+:::{tip}
+Refer to [{{esql}} for search](docs-content://solutions/search/esql-for-search.md#esql-for-search-scoring) for more information on relevance scoring and how to use `_score` in your queries.
+:::
diff --git a/docs/reference/query-languages/esql/functions-operators/search-functions.md b/docs/reference/query-languages/esql/functions-operators/search-functions.md
@@ -6,21 +6,27 @@ mapped_pages:
 
 # {{esql}} Search functions [esql-search-functions]
 
+:::{tip}
+Get started with {{esql}} for search use cases with
+our [hands-on tutorial](docs-content://solutions/search/esql-search-tutorial.md).
+
+For a high-level overview of search functionalities in {{esql}}, and to learn about relevance scoring, refer to [{{esql}} for search](docs-content://solutions/search/esql-for-search.md#esql-for-search-scoring).
+:::
+
+{{esql}} provides a set of functions for performing searching on text fields. 
+
 Use these functions
 for [full-text search](docs-content://solutions/search/full-text.md)
 and [semantic search](docs-content://solutions/search/semantic-search/semantic-search-semantic-text.md).
 
-Get started with {{esql}} for search use cases with
-our [hands-on tutorial](docs-content://solutions/search/esql-search-tutorial.md).
-
 Full text functions can be used to
 match [multivalued fields](/reference/query-languages/esql/esql-multivalued-fields.md).
 A multivalued field that contains a value that matches a full text query is
 considered to match the query.
 
 Full text functions are significantly more performant for text search use cases
 on large data sets than using pattern matching or regular expressions with
-`LIKE` or `RLIKE`
+`LIKE` or `RLIKE`.
 
 See [full text search limitations](/reference/query-languages/esql/limitations.md#esql-limitations-full-text-search)
 for information on the limitations of full text search.
diff --git a/docs/release-notes/breaking-changes.md b/docs/release-notes/breaking-changes.md
@@ -12,23 +12,15 @@ If you are migrating from a version prior to version 9.0, you must first upgrade
 
 % ## Next version [elasticsearch-nextversion-breaking-changes]
 
-```{applies_to}
-stack: coming 9.0.3
-```
 ## 9.0.3 [elasticsearch-9.0.3-breaking-changes]
 
 No breaking changes in this version.
 
-```{applies_to}
-stack: coming 9.0.2
-```
 ## 9.0.2 [elasticsearch-9.0.2-breaking-changes]
 
 Snapshot/Restore:
 * Make S3 custom query parameter optional [#128043](https://github.com/elastic/elasticsearch/pull/128043)
 
-
-
 ## 9.0.1 [elasticsearch-9.0.1-breaking-changes]
 
 No breaking changes in this version.
diff --git a/docs/release-notes/changelog-bundles/9.0.2.yml b/docs/release-notes/changelog-bundles/9.0.2.yml
@@ -1,6 +1,6 @@
 version: 9.0.2
-released: false
-generated: 2025-05-22T15:14:00.768080Z
+released: true
+generated: 2025-06-03T14:58:36.937148609Z
 changelogs:
   - pr: 126992
     summary: Add missing `outbound_network` entitlement to x-pack-core
@@ -36,6 +36,12 @@ changelogs:
     type: bug
     issues:
       - 127468
+  - pr: 127564
+    summary: Consider inlinestats when having `field_caps` check for field names
+    area: ES|QL
+    type: bug
+    issues:
+      - 127236
   - pr: 127658
     summary: Append all data to Chat Completion buffer
     area: Machine Learning
@@ -133,3 +139,31 @@ changelogs:
     area: Data streams
     type: bug
     issues: []
+  - pr: 128259
+    summary: Added geometry validation for GEO types to exit early on invalid latitudes
+    area: Geo
+    type: bug
+    issues:
+      - 128234
+  - pr: 128260
+    summary: Fix validation NPE in Enrich and add extra @Nullable annotations
+    area: ES|QL
+    type: bug
+    issues:
+      - 126297
+      - 126253
+  - pr: 128320
+    summary: Use new source loader when lower `docId` is accessed
+    area: Codec
+    type: bug
+    issues: []
+  - pr: 128327
+    summary: Use internal user for internal inference action
+    area: Machine Learning
+    type: bug
+    issues: []
+  - pr: 128338
+    summary: Pass timeout to chat completion
+    area: Machine Learning
+    type: bug
+    issues: []
diff --git a/docs/release-notes/changelog-bundles/9.0.3.yml b/docs/release-notes/changelog-bundles/9.0.3.yml
@@ -1,6 +1,6 @@
 version: 9.0.3
-released: false
-generated: 2025-06-21T00:06:16.346021604Z
+released: true
+generated: 2025-06-24T15:19:29.859630035Z
 changelogs:
   - pr: 120869
     summary: Threadpool merge scheduler
diff --git a/docs/release-notes/deprecations.md b/docs/release-notes/deprecations.md
@@ -16,19 +16,11 @@ To give you insight into what deprecated features you’re using, {{es}}:
 
 % ## Next version [elasticsearch-nextversion-deprecations]
 
-```{applies_to}
-stack: coming 9.0.3
-```
 ## 9.0.3 [elasticsearch-9.0.3-deprecations]
 
 Engine:
 * Deprecate `indices.merge.scheduler.use_thread_pool` setting [#129464](https://github.com/elastic/elasticsearch/pull/129464)
 
-
-
-```{applies_to}
-stack: coming 9.0.2
-```
 ## 9.0.2 [elasticsearch-9.0.2-deprecations]
 
 No deprecations in this version.
diff --git a/docs/release-notes/index.md b/docs/release-notes/index.md
@@ -21,9 +21,6 @@ To check for security updates, go to [Security announcements for the Elastic sta
 % *
 
 ## 9.0.3 [elasticsearch-9.0.3-release-notes]
-```{applies_to}
-stack: coming 9.0.3
-```
 
 ### Features and enhancements [elasticsearch-9.0.3-features-enhancements]
 
@@ -92,11 +89,7 @@ Searchable Snapshots:
 Security:
 * Fix error message when changing the password for a user in the file realm [#127621](https://github.com/elastic/elasticsearch/pull/127621)
 
-
 ## 9.0.2 [elasticsearch-9.0.2-release-notes]
-```{applies_to}
-stack: coming 9.0.2
-```
 
 ### Features and enhancements [elasticsearch-9.0.2-features-enhancements]
 
@@ -114,19 +107,27 @@ Aggregations:
 Audit:
 * Handle streaming request body in audit log [#127798](https://github.com/elastic/elasticsearch/pull/127798)
 
+Codec:
+* Use new source loader when lower `docId` is accessed [#128320](https://github.com/elastic/elasticsearch/pull/128320)
+
 Data streams:
 * Fix system data streams incorrectly showing up in the list of template validation problems [#128161](https://github.com/elastic/elasticsearch/pull/128161)
 
 Downsampling:
 * Downsampling does not consider passthrough fields as dimensions [#127752](https://github.com/elastic/elasticsearch/pull/127752) (issue: [#125156](https://github.com/elastic/elasticsearch/issues/125156))
 
 ES|QL:
+* Consider inlinestats when having `field_caps` check for field names [#127564](https://github.com/elastic/elasticsearch/pull/127564) (issue: [#127236](https://github.com/elastic/elasticsearch/issues/127236))
 * Don't push down filters on the right hand side of an inlinejoin [#127383](https://github.com/elastic/elasticsearch/pull/127383)
 * ESQL: Avoid unintended attribute removal [#127563](https://github.com/elastic/elasticsearch/pull/127563) (issue: [#127468](https://github.com/elastic/elasticsearch/issues/127468))
 * ESQL: Fix alias removal in regex extraction with JOIN [#127687](https://github.com/elastic/elasticsearch/pull/127687) (issue: [#127467](https://github.com/elastic/elasticsearch/issues/127467))
 * ESQL: Keep `DROP` attributes when resolving field names [#127009](https://github.com/elastic/elasticsearch/pull/127009) (issue: [#126418](https://github.com/elastic/elasticsearch/issues/126418))
 * Ensure ordinal builder emit ordinal blocks [#127949](https://github.com/elastic/elasticsearch/pull/127949)
 * Fix union types in CCS [#128111](https://github.com/elastic/elasticsearch/pull/128111)
+* Fix validation NPE in Enrich and add extra @Nullable annotations [#128260](https://github.com/elastic/elasticsearch/pull/128260) (issues: [#126297](https://github.com/elastic/elasticsearch/issues/126297), [#126253](https://github.com/elastic/elasticsearch/issues/126253))
+
+Geo:
+* Added geometry validation for GEO types to exit early on invalid latitudes [#128259](https://github.com/elastic/elasticsearch/pull/128259) (issue: [#128234](https://github.com/elastic/elasticsearch/issues/128234))
 
 Infra/Core:
 * Add missing `outbound_network` entitlement to x-pack-core [#126992](https://github.com/elastic/elasticsearch/pull/126992) (issue: [#127003](https://github.com/elastic/elasticsearch/issues/127003))
@@ -138,6 +139,8 @@ Infra/Scripting:
 Machine Learning:
 * Append all data to Chat Completion buffer [#127658](https://github.com/elastic/elasticsearch/pull/127658)
 * Fix services API Google Vertex AI Rerank location field requirement [#127856](https://github.com/elastic/elasticsearch/pull/127856)
+* Pass timeout to chat completion [#128338](https://github.com/elastic/elasticsearch/pull/128338)
+* Use internal user for internal inference action [#128327](https://github.com/elastic/elasticsearch/pull/128327)
 
 Relevance:
 * Fix: Add `NamedWriteable` for `RuleQueryRankDoc` [#128153](https://github.com/elastic/elasticsearch/pull/128153) (issue: [#126071](https://github.com/elastic/elasticsearch/issues/126071))
diff --git a/libs/core/src/main/java/org/elasticsearch/core/NotMultiProjectCapable.java b/libs/core/src/main/java/org/elasticsearch/core/NotMultiProjectCapable.java
@@ -0,0 +1,34 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+package org.elasticsearch.core;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Annotation to identify a block of code (a whole class, a method, a field, or a local variable) that is intentionally not fully
+ * project-aware because it's not intended to be used in a serverless environment. Some features are unavailable in serverless and are
+ * thus not worth the investment to make fully project-aware. This annotation makes it easier to identify blocks of code that require
+ * attention in case those features are revisited from a multi-project POV.
+ */
+@Retention(RetentionPolicy.SOURCE)
+@Target(
+    { ElementType.LOCAL_VARIABLE, ElementType.CONSTRUCTOR, ElementType.FIELD, ElementType.METHOD, ElementType.TYPE, ElementType.MODULE }
+)
+public @interface NotMultiProjectCapable {
+
+    /**
+     * Some explanation on why the block of code would not work in a multi-project context and/or what would need to be done to make it
+     * properly project-aware.
+     */
+    String description() default "";
+}
diff --git a/muted-tests.yml b/muted-tests.yml
@@ -561,7 +561,12 @@ tests:
   method: "builds distribution from branches via archives extractedAssemble [bwcDistVersion: 8.2.1, bwcProject: bugfix, expectedAssembleTaskName:
     extractedAssemble, #2]"
   issue: https://github.com/elastic/elasticsearch/issues/119871
-
+- class: org.elasticsearch.xpack.inference.qa.mixed.CohereServiceMixedIT
+  method: testRerank
+  issue: https://github.com/elastic/elasticsearch/issues/130009
+- class: org.elasticsearch.xpack.inference.qa.mixed.CohereServiceMixedIT
+  method: testCohereEmbeddings
+  issue: https://github.com/elastic/elasticsearch/issues/130010
 
 # Examples:
 #
diff --git a/test/fixtures/minio-fixture/src/main/java/org/elasticsearch/test/fixtures/minio/MinioTestContainer.java b/test/fixtures/minio-fixture/src/main/java/org/elasticsearch/test/fixtures/minio/MinioTestContainer.java
@@ -17,7 +17,7 @@ public final class MinioTestContainer extends DockerEnvironmentAwareTestContaine
 
     // NB releases earlier than 2025-05-24 are buggy, see https://github.com/minio/minio/issues/21189, and #127166 for a workaround
     // However the 2025-05-24 release is also buggy, see https://github.com/minio/minio/issues/21377, and this has no workaround
-    public static final String DOCKER_BASE_IMAGE = "minio/minio:RELEASE.2024-12-18T13-15-44Z";
+    public static final String DOCKER_BASE_IMAGE = "minio/minio:RELEASE.2025-06-13T11-33-47Z";
 
     private static final int servicePort = 9000;
     private final boolean enabled;
diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/EsPhysicalOperationProviders.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/EsPhysicalOperationProviders.java
diff --git a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/planner/LocalExecutionPlannerTests.java b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/planner/LocalExecutionPlannerTests.java
diff --git a/x-pack/plugin/logsdb/src/yamlRestTest/resources/rest-api-spec/test/52_esql_insist_operator_synthetic_source.yml b/x-pack/plugin/logsdb/src/yamlRestTest/resources/rest-api-spec/test/52_esql_insist_operator_synthetic_source.yml
diff --git a/x-pack/plugin/snapshot-repo-test-kit/qa/minio/src/javaRestTest/java/org/elasticsearch/repositories/blobstore/testkit/analyze/MinioRepositoryAnalysisRestIT.java b/x-pack/plugin/snapshot-repo-test-kit/qa/minio/src/javaRestTest/java/org/elasticsearch/repositories/blobstore/testkit/analyze/MinioRepositoryAnalysisRestIT.java
