Merge remote-tracking branch 'upstream/main' into index-pipeline-tracking

* upstream/main: (33 commits)
  Allow both WithEntitlementsOnTestCode and EntitledTestPackages together (elastic#130826)
  Move streams status actions to cluster:monitor group (elastic#131015)
  Update JDK base image for OIDC fixture (elastic#131176)
  Mute org.elasticsearch.xpack.esql.ccq.MultiClustersIT testLookupJoinAliases elastic#131166
  Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testEnqueuedMergeTasksAreUnblockedWhenEstimatedMergeSizeChanges elastic#131165
  Mute org.elasticsearch.xpack.esql.ccq.MultiClustersIT testNotLikeListKeyword elastic#131155
  Mute org.elasticsearch.xpack.esql.qa.multi_node.GenerativeIT test elastic#131154
  Check file entitlements on the Lucene FilterFileSystem in tests (elastic#130825)
  Mute org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT test {lookup-join.MvJoinKeyOnFromAfterStats ASYNC} elastic#131148
  Move FrequencyCappedAction to common package (elastic#131060)
  Mute org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryStopIT testStopQueryLocal elastic#121672
  Remove nesting from multi allocation decision (elastic#130844)
  Disable async search rest tests in release builds (elastic#131132)
  Fix testStopQueryLocal (elastic#131130)
  Fixes based on resharding disruption tests (elastic#130870)
  Remove inactive logger (elastic#131121)
  Add wait for remote start for the test (elastic#131124)
  Add existing shards allocator settings to failure store allowed list. (elastic#131056)
  Don't allow field caps to use semantic queries as index filters (elastic#131111)
  issue should be already fixed by elastic#121466 (elastic#130860)
  ...

Author: szybia

docs/changelog/131015.yaml

@@ -0,0 +1,5 @@
+pr: 131015
+summary: Move streams status actions to cluster:monitor group
+area: Data streams
+type: bug
+issues: []

docs/changelog/131032.yaml

@@ -0,0 +1,5 @@
+pr: 131032
+summary: "Fix: `GET _synonyms` returns synonyms with empty rules"
+area: Relevance
+type: bug
+issues: []

docs/changelog/131056.yaml

@@ -0,0 +1,5 @@
+pr: 131056
+summary: Add existing shards allocator settings to failure store allowed list
+area: Data streams
+type: bug
+issues: []

docs/changelog/131081.yaml

@@ -0,0 +1,6 @@
+pr: 131081
+summary: Fix knn search error when dimensions are not set
+area: Vector Search
+type: bug
+issues:
+ - 129550

docs/changelog/131111.yaml

@@ -0,0 +1,5 @@
+pr: 131111
+summary: Don't allow field caps to use semantic queries as index filters
+area: Search
+type: bug
+issues: []

docs/reference/query-languages/esql/limitations.md

@@ -250,3 +250,6 @@ Work around this limitation by converting the field to single value with one of
 * CSV export from Discover shows no more than 10,000 rows. This limit only applies to the number of rows that are retrieved by the query and displayed in Discover. Queries and aggregations run on the full data set.
 * Querying many indices at once without any filters can cause an error in kibana which looks like `[esql] > Unexpected error from Elasticsearch: The content length (536885793) is bigger than the maximum allowed string (536870888)`. The response from {{esql}} is too long. Use [`DROP`](/reference/query-languages/esql/commands/processing-commands.md#esql-drop) or [`KEEP`](/reference/query-languages/esql/commands/processing-commands.md#esql-keep) to limit the number of fields returned.
 
+## Known issues [esql-known-issues]
+
+Refer to [Known issues](/release-notes/known-issues.md) for a list of known issues for {{esql}}.

docs/reference/query-languages/query-dsl/query-dsl-bool-query.md

@@ -13,8 +13,8 @@ A query that matches documents matching boolean combinations of other queries. T
 | --- | --- |
 | `must` | The clause (query) must appear in matching documents and will contribute to the score. Each query defined under a `must` acts as a logical "AND", returning only documents that match *all* the specified queries. |
 | `should` | The clause (query) should appear in the matching document. Each query defined under a `should` acts as a logical "OR", returning documents that match *any* of the specified queries. |
-| `filter` | The clause (query) must appear in matching documents. However unlike`must` the score of the query will be ignored. Filter clauses are executedin [filter context](/reference/query-languages/query-dsl/query-filter-context.md), meaning that scoring is ignoredand clauses are considered for caching. Each query defined under a `filter` acts as a logical "AND", returning only documents that match *all* the specified queries. |
-| `must_not` | The clause (query) must not appear in the matchingdocuments. Clauses are executed in [filter context](/reference/query-languages/query-dsl/query-filter-context.md) meaningthat scoring is ignored and clauses are considered for caching. Because scoring isignored, a score of `0` for all documents is returned. Each query defined under a `must_not` acts as a logical "NOT", returning only documents that do not match any of the specified queries. |
+| `filter` | The clause (query) must appear in matching documents. However unlike `must` the score of the query will be ignored. Filter clauses are executed in [filter context](/reference/query-languages/query-dsl/query-filter-context.md), meaning that scoring is ignored and clauses are considered for caching. Each query defined under a `filter` acts as a logical "AND", returning only documents that match *all* the specified queries. |
+| `must_not` | The clause (query) must not appear in the matching documents. Clauses are executed in [filter context](/reference/query-languages/query-dsl/query-filter-context.md) meaning that scoring is ignored and clauses are considered for caching. Because scoring is ignored, a score of `0` for all documents is returned. Each query defined under a `must_not` acts as a logical "NOT", returning only documents that do not match any of the specified queries. |
 
 The `must` and `should` clauses function as logical AND, OR operators, contributing to the scoring of results. However, these results are not cached, which means repeated queries won't benefit from faster retrieval. In contrast, the `filter` and `must_not` clauses are used to include or exclude results without impacting the score, unless used within a `constant_score` query.
 

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md

@@ -165,7 +165,6 @@ POST my-image-index/_search
 
 Knn query can be used as a part of hybrid search, where knn query is combined with other lexical queries. For example, the query below finds documents with `title` matching `mountain lake`, and combines them with the top 10 documents that have the closest image vectors to the `query_vector`. The combined documents are then scored and the top 3 top scored documents are returned.
 
-+
 
 ```console
 POST my-image-index/_search

docs/release-notes/known-issues.md

@@ -8,10 +8,19 @@ mapped_pages:
 Known issues are significant defects or limitations that may impact your implementation. These issues are actively being worked on and will be addressed in a future release. Review the Elasticsearch known issues to help you make informed decisions, such as upgrading to a new version.
 
 ## 9.0.3 [elasticsearch-9.0.3-known-issues]
-A bug in the merge scheduler in Elasticsearch 9.0.3 may prevent shards from closing when there isn’t enough disk space to complete a merge. As a result, operations such as closing or relocating an index may hang until sufficient disk space becomes available.
+* A bug in the merge scheduler in Elasticsearch 9.0.3 may prevent shards from closing when there isn’t enough disk space to complete a merge. As a result, operations such as closing or relocating an index may hang until sufficient disk space becomes available.
 To mitigate this issue, the disk space checker is disabled by default in 9.0.3 by setting `indices.merge.disk.check_interval` to `0` seconds. Manually enabling this setting is not recommended.
 
-This issue is planned to be fixed in future patch release [#129613](https://github.com/elastic/elasticsearch/pull/129613)
+  This issue is planned to be fixed in future patch release [#129613](https://github.com/elastic/elasticsearch/pull/129613)
+
+* A bug in the ES|QL STATS command may yield incorrect results. The bug only happens in very specific cases that follow this pattern: `STATS ... BY keyword1, keyword2`, i.e. the command must have exactly two grouping fields, both keywords, where the first field has high cardinality (more than 65k distinct values).
+
+  The bug is described in detail in [this issue](https://github.com/elastic/elasticsearch/issues/130644).
+  The problem was introduced in 8.16.0 and [fixed](https://github.com/elastic/elasticsearch/pull/130705) in 8.17.9, 8.18.7, 9.0.4.
+
+  Possible workarounds include:
+  * switching the order of the grouping keys (eg. `STATS ... BY keyword2, keyword1`, if the `keyword2` has a lower cardinality)
+  * reducing the grouping key cardinality, by filtering out values before STATS
 
 ## 9.0.0 [elasticsearch-9.0.0-known-issues]
 * Elasticsearch on Windows might fail to start, or might forbid some file-related operations, when referencing paths with a case different from the one stored by the filesystem. Windows treats paths as case-insensitive, but the filesystem stores them with case. Entitlements, the new security system used by Elasticsearch, treat all paths as case-sensitive, and can therefore prevent access to a path that should be accessible.
@@ -40,3 +49,12 @@ This issue will be fixed in a future patch release (see [PR #126990](https://git
      DELETE _index_template/.watches
      POST /_watcher/_start
      ```
+
+* A bug in the ES|QL STATS command may yield incorrect results. The bug only happens in very specific cases that follow this pattern: `STATS ... BY keyword1, keyword2`, i.e. the command must have exactly two grouping fields, both keywords, where the first field has high cardinality (more than 65k distinct values).
+
+  The bug is described in detail in [this issue](https://github.com/elastic/elasticsearch/issues/130644).
+  The problem was introduced in 8.16.0 and [fixed](https://github.com/elastic/elasticsearch/pull/130705) in 8.17.9, 8.18.7, 9.0.4.
+
+  Possible workarounds include:
+    * switching the order of the grouping keys (eg. `STATS ... BY keyword2, keyword1`, if the `keyword2` has a lower cardinality)
+    * reducing the grouping key cardinality, by filtering out values before STATS

libs/entitlement/src/main/java/org/elasticsearch/entitlement/bootstrap/EntitlementBootstrap.java

@@ -161,7 +161,7 @@ private static PolicyManager createPolicyManager(
         PathLookup pathLookup,
         Policy serverPolicyPatch,
         Function<Class<?>, PolicyManager.PolicyScope> scopeResolver,
-        Map<String, Collection<Path>> pluginSourcePaths
+        Map<String, Collection<Path>> pluginSourcePathsResolver
     ) {
         FilesEntitlementsValidation.validate(pluginPolicies, pathLookup);
 
@@ -170,7 +170,7 @@ private static PolicyManager createPolicyManager(
             HardcodedEntitlements.agentEntitlements(),
             pluginPolicies,
             scopeResolver,
-            pluginSourcePaths,
+            pluginSourcePathsResolver::get,
             pathLookup
         );
     }