Merge remote-tracking branch 'upstream/main' into security-stats

* upstream/main:
  Add additional logging to make spotting stats issues easier (elastic#133972)
  [ESQL] Clean up ESQL enrich landing page (elastic#134820)
  ES|QL: Make kibana docs for Query settings more consistent (elastic#134881)
  Add file extension metadata to cache miss counter from SharedBlobCacheService (elastic#134374)
  Add IT for num_reduced_phases with batched query execution (elastic#134312)
  Remove `SizeValue` (elastic#134871)

Author: szybia

docs/changelog/134374.yaml

@@ -0,0 +1,5 @@
+pr: 134374
+summary: Add file extension metadata to cache miss counter from `SharedBlobCacheService`
+area: Search
+type: enhancement
+issues: []

docs/reference/query-languages/esql/esql-enrich-data.md

@@ -1,29 +1,38 @@
 ---
+applies_to:
+  stack:
+  serverless: unavailable
 navigation_title: "Combine data with ENRICH"
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-enrich-data.html
 ---
 
 # Combine data from multiple indices with `ENRICH` [esql-enrich-data]
 
+This page provides an overview of the {{esql}} `ENRICH` command. For complete syntax details and examples, refer to the [`ENRICH` command reference](/reference/query-languages/esql/commands/enrich.md).
+
 The {{esql}} [`ENRICH`](/reference/query-languages/esql/commands/enrich.md) processing command combines, at query-time, data from one or more source indexes with field-value combinations found in {{es}} enrich indexes.
 
 For example, you can use `ENRICH` to:
-
 * Identify web services or vendors based on known IP addresses
 * Add product information to retail orders based on product IDs
 * Supplement contact information based on an email address
 
-[`ENRICH`](/reference/query-languages/esql/commands/enrich.md) is similar to [`LOOKUP join`](/reference/query-languages/esql/commands/lookup-join.md) in the fact that they both help you join data together. You should use `ENRICH` when:
+## Compare `ENRICH` and `LOOKUP JOIN`
 
+[`ENRICH`](/reference/query-languages/esql/commands/enrich.md) is similar to [`LOOKUP join`](/reference/query-languages/esql/commands/lookup-join.md) in the fact that they both help you join data together. You should use `ENRICH` when:
 * Enrichment data doesn't change frequently
 * You can accept index-time overhead
 * You can accept having multiple matches combined into multi-values
 * You can accept being limited to predefined match fields
 * You do not need fine-grained security: There are no restrictions to specific enrich policies or document and field level security.
 * You want to match using ranges or spatial relations
 
-### How the `ENRICH` command works [esql-how-enrich-works]
+## Syntax reference
+
+For complete syntax details and examples, refer to the [ENRICH command reference](/reference/query-languages/esql/commands/enrich.md).
+
+## How the `ENRICH` command works [esql-how-enrich-works]
 
 The `ENRICH` command adds new columns to a table, with data from {{es}} indices. It requires a few special components:
 
@@ -43,8 +52,7 @@ An enrich policy contains:
 * A *match field* from the source indices used to match incoming documents
 * *Enrich fields* containing enrich data from the source indices you want to add to incoming documents
 
-After [creating a policy](#esql-create-enrich-policy), it must be [executed](#esql-execute-enrich-policy) before it can be used. Executing an enrich policy uses data from the policy’s source indices to create a streamlined system index called the *enrich index*. The `ENRICH` command uses this index to match and enrich an input table.
-
+After [creating a policy](#esql-create-enrich-policy), it must be [executed](#esql-execute-enrich-policy) before it can be used. Executing an enrich policy uses data from the policy's source indices to create a streamlined system index called the *enrich index*. The `ENRICH` command uses this index to match and enrich an input table.
 
 $$$esql-source-index$$$
 
@@ -60,14 +68,16 @@ Directly matching rows from input tables to documents in source indices could be
 
 Enrich indices contain enrich data from source indices but have a few special properties to help streamline them:
 
-* They are system indices, meaning they’re managed internally by {{es}} and only intended for use with enrich processors and the {{esql}} `ENRICH` command.
+* They are system indices, meaning they're managed internally by {{es}} and only intended for use with enrich processors and the {{esql}} `ENRICH` command.
 * They always begin with `.enrich-*`.
-* They are read-only, meaning you can’t directly change them.
+* They are read-only, meaning you can't directly change them.
 * They are [force merged](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) for fast retrieval.
 
+## Using `ENRICH` across clusters
 
+You can use `ENRICH` with remote clusters. For detailed information about cross-cluster enrichment syntax and configuration, refer to [ENRICH across clusters](/reference/query-languages/esql/esql-cross-clusters.md#ccq-enrich).
 
-### Set up an enrich policy [esql-set-up-enrich-policy]
+## Set up an enrich policy [esql-set-up-enrich-policy]
 
 To start using `ENRICH`, follow these steps:
 
@@ -81,19 +91,15 @@ Once you have enrich policies set up, you can [update your enrich data](#esql-up
 
 ::::{important}
 The `ENRICH` command performs several operations and may impact the speed of your query.
-
 ::::
 
-
-
 ### Prerequisites [esql-enrich-prereqs]
 
 To use enrich policies, you must have:
 
 * `read` index privileges for any indices used
 * The `enrich_user` [built-in role](/reference/elasticsearch/roles.md)
 
-
 ### Add enrich data [esql-create-enrich-source-index]
 
 To begin, add documents to one or more source indices. These documents should contain the enrich data you eventually want to add to incoming data.
@@ -102,18 +108,14 @@ You can manage source indices just like regular {{es}} indices using the [docume
 
 You also can set up [{{beats}}](beats://reference/index.md), such as a [{{filebeat}}](beats://reference/filebeat/filebeat-installation-configuration.md), to automatically send and index documents to your source indices. See [Getting started with {{beats}}](beats://reference/index.md).
 
-
 ### Create an enrich policy [esql-create-enrich-policy]
 
 After adding enrich data to your source indices, use the [create enrich policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-put-policy) or [Index Management in {{kib}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html#manage-enrich-policies) to create an enrich policy.
 
 ::::{warning}
-Once created, you can’t update or change an enrich policy. See [Update an enrich policy](docs-content://manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md#update-enrich-policies).
-
+Once created, you can't update or change an enrich policy. See [Update an enrich policy](docs-content://manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md#update-enrich-policies).
 ::::
 
-
-
 ### Execute the enrich policy [esql-execute-enrich-policy]
 
 Once the enrich policy is created, you need to execute it using the [execute enrich policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy) or [Index Management in {{kib}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html#manage-enrich-policies) to create an [enrich index](docs-content://manage-data/ingest/transform-enrich/data-enrichment.md#enrich-index).
@@ -122,15 +124,13 @@ Once the enrich policy is created, you need to execute it using the [execute enr
 :alt: esql enrich policy
 :::
 
-The *enrich index* contains documents from the policy’s source indices. Enrich indices always begin with `.enrich-*`, are read-only, and are [force merged](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge).
+The *enrich index* contains documents from the policy's source indices. Enrich indices always begin with `.enrich-*`, are read-only, and are [force merged](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge).
 
 ::::{warning}
 Enrich indices should only be used by the [enrich processor](/reference/enrich-processor/enrich-processor.md) or the [{{esql}} `ENRICH` command](/reference/query-languages/esql/commands/enrich.md). Avoid using enrich indices for other purposes.
 
 ::::
 
-
-
 ### Use the enrich policy [esql-use-enrich]
 
 After the policy has been executed, you can use the [`ENRICH` command](/reference/query-languages/esql/commands/enrich.md) to enrich your data.
@@ -185,15 +185,13 @@ ROW a = "1"
 
 In case of name collisions, the newly created columns will override existing columns.
 
-
-### Update an enrich index [esql-update-enrich-data]
+## Update an enrich index [esql-update-enrich-data]
 
 Once created, you cannot update or index documents to an enrich index. Instead, update your source indices and [execute](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy) the enrich policy again. This creates a new enrich index from your updated source indices. The previous enrich index will be deleted with a delayed maintenance job that executes by default every 15 minutes.
 
+## Update an enrich policy [esql-update-enrich-policies]
 
-### Update an enrich policy [esql-update-enrich-policies]
-
-Once created, you can’t update or change an enrich policy. Instead, you can:
+Once created, you can't update or change an enrich policy. Instead, you can:
 
 1. Create and [execute](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy) a new enrich policy.
 2. Replace the previous enrich policy with the new enrich policy in any in-use enrich processors or {{esql}} queries.
@@ -204,18 +202,22 @@ Once created, you can’t update or change an enrich policy. Instead, you can:
 The {{esql}} `ENRICH` command supports all three enrich policy types:
 
 `geo_match`
-:   Matches enrich data to incoming documents based on a [`geo_shape` query](/reference/query-languages/query-dsl/query-dsl-geo-shape-query.md). For an example, see [Example: Enrich your data based on geolocation](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md).
+:   Matches enrich data to incoming documents based on a [`geo_shape` query](/reference/query-languages/query-dsl/query-dsl-geo-shape-query.md). For an example, refer to [Example: Enrich your data based on geolocation](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md).
 
 `match`
-:   Matches enrich data to incoming documents based on a [`term` query](/reference/query-languages/query-dsl/query-dsl-term-query.md). For an example, see [Example: Enrich your data based on exact values](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md).
+:   Matches enrich data to incoming documents based on a [`term` query](/reference/query-languages/query-dsl/query-dsl-term-query.md). For an example, refer to [Example: Enrich your data based on exact values](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md).
 
 `range`
-:   Matches a number, date, or IP address in incoming documents to a range in the enrich index based on a [`term` query](/reference/query-languages/query-dsl/query-dsl-term-query.md). For an example, see [Example: Enrich your data by matching a value to a range](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md).
+:   Matches a number, date, or IP address in incoming documents to a range in the enrich index based on a [`term` query](/reference/query-languages/query-dsl/query-dsl-term-query.md). For an example, refer to [Example: Enrich your data by matching a value to a range](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md).
 
 While all three enrich policy types are supported, there are some limitations to be aware of:
 
 * The `geo_match` enrich policy type only supports the `intersects` spatial relation.
 * It is required that the `match_field` in the `ENRICH` command is of the correct type. For example, if the enrich policy is of type `geo_match`, the `match_field` in the `ENRICH` command must be of type `geo_point` or `geo_shape`. Likewise, a `range` enrich policy requires a `match_field` of type `integer`, `long`, `date`, or `ip`, depending on the type of the range field in the original enrich index.
 * However, this constraint is relaxed for `range` policies when the `match_field` is of type `KEYWORD`. In this case the field values will be parsed during query execution, row by row. If any value fails to parse, the output values for that row will be set to `null`, an appropriate warning will be produced and the query will continue to execute.
 
-
+## Related pages
+  
+* [`ENRICH` command reference](/reference/query-languages/esql/commands/enrich.md): Complete syntax documentation and examples
+* [`ENRICH` across clusters](/reference/query-languages/esql/esql-cross-clusters.md#ccq-enrich) - Cross-cluster enrichment configuration
+* [LOOKUP JOIN command](/reference/query-languages/esql/commands/lookup-join.md) - Alternative approach for joining data

docs/reference/query-languages/esql/kibana/definition/settings/project_routing.json

@@ -233,7 +233,7 @@ private static void blockThreadPool(String threadPoolName, ThreadPool threadPool
     private static void fillThreadPoolQueues(String threadPoolName, ThreadPool threadPool) {
         ThreadPool.Info info = threadPool.info(threadPoolName);
 
-        for (int i = 0; i < info.getQueueSize().singles(); i++) {
+        for (int i = 0; i < info.getQueueSize(); i++) {
             try {
                 threadPool.executor(threadPoolName).execute(() -> {});
             } catch (EsRejectedExecutionException e) {

modules/kibana/src/internalClusterTest/java/org/elasticsearch/kibana/KibanaThreadPoolIT.java

@@ -9,10 +9,13 @@
 
 package org.elasticsearch.index.store;
 
+import org.apache.lucene.index.IndexFileNames;
 import org.elasticsearch.core.Assertions;
 import org.elasticsearch.core.SuppressForbidden;
 import org.elasticsearch.test.ESTestCase;
 
+import java.util.Locale;
+
 import static org.hamcrest.Matchers.containsString;
 
 public class LuceneFilesExtensionsTests extends ESTestCase {
@@ -21,6 +24,7 @@ public void testUnknownFileExtension() {
         if (Assertions.ENABLED) {
             AssertionError e = expectThrows(AssertionError.class, () -> LuceneFilesExtensions.fromExtension("abc"));
             assertThat(e.getMessage(), containsString("unknown Lucene file extension [abc]"));
+            assertFalse(LuceneFilesExtensions.isLuceneExtension("abc"));
 
             setEsAllowUnknownLuceneFileExtensions("true");
             try {
@@ -41,4 +45,19 @@ public void setEsAllowUnknownLuceneFileExtensions(final String value) {
             System.setProperty("es.allow_unknown_lucene_file_extensions", value);
         }
     }
+
+    public void testIsLuceneExtension() {
+        assertFalse(LuceneFilesExtensions.isLuceneExtension(null));
+        assertFalse(LuceneFilesExtensions.isLuceneExtension("bcde"));
+        String randomStringWithLuceneExtension = randomAlphanumericOfLength(10)
+            + "."
+            + LuceneFilesExtensions.values()[randomInt(LuceneFilesExtensions.values().length) - 1].getExtension();
+        String extension = IndexFileNames.getExtension(randomStringWithLuceneExtension);
+        assertTrue(extension + " should be considered a Lucene extension", LuceneFilesExtensions.isLuceneExtension(extension));
+        String upperCaseExtension = extension.toUpperCase(Locale.ROOT);
+        assertFalse(
+            upperCaseExtension + " (uppercase) should not be considered a Lucene extension",
+            LuceneFilesExtensions.isLuceneExtension(upperCaseExtension)
+        );
+    }
 }

qa/evil-tests/src/test/java/org/elasticsearch/index/store/LuceneFilesExtensionsTests.java

@@ -1,7 +1,4 @@
 setup:
-  - skip:
-      awaits_fix: "TODO fix this test, the response with batched execution is not deterministic enough for the available matchers"
-
   - do:
       indices.create:
           index: test_1

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search/120_batch_reduce_size.yml

@@ -558,7 +558,7 @@ public boolean isForceExecution() {
     }
 
     private static void fillWriteCoordinationQueue(ThreadPool threadPool) {
-        final var queueSize = Math.toIntExact(threadPool.info(ThreadPool.Names.WRITE_COORDINATION).getQueueSize().singles());
+        final var queueSize = Math.toIntExact(threadPool.info(ThreadPool.Names.WRITE_COORDINATION).getQueueSize());
         final var queueFilled = new AtomicBoolean(false);
         final var queueFillingTask = new AbstractRunnable() {
             @Override