Merge branch 'main' into remove-bulk-processor

Author: nielsbauman

docs/changelog/129089.yaml

@@ -0,0 +1,5 @@
+pr: 129089
+summary: Update `sparse_vector` field mapping to include default setting for token pruning
+area: Mapping
+type: enhancement
+issues: []

docs/changelog/129413.yaml

@@ -0,0 +1,5 @@
+pr: 129413
+summary: '`SageMaker` Elastic Payload'
+area: Machine Learning
+type: enhancement
+issues: []

docs/changelog/129557.yaml

@@ -0,0 +1,5 @@
+pr: 129557
+summary: Pushdown for LIKE (LIST)
+area: ES|QL
+type: enhancement
+issues: []

docs/reference/elasticsearch/mapping-reference/sparse-vector.md

@@ -24,6 +24,33 @@ PUT my-index
 }
 ```
 
+## Token pruning
+```{applies_to}
+stack: preview 9.1
+```
+
+With any new indices created, token pruning will be turned on by default with appropriate defaults. You can control this behaviour using the optional `index_options` parameters for the field:
+
+```console
+PUT my-index
+{
+  "mappings": {
+    "properties": {
+      "text.tokens": {
+        "type": "sparse_vector",
+        "index_options": {
+          "prune": true,
+          "pruning_config": {
+            "tokens_freq_ratio_threshold": 5,
+            "tokens_weight_threshold": 0.4
+          }
+        }
+      }
+    }
+  }
+}
+```
+
 See [semantic search with ELSER](docs-content://solutions/search/semantic-search/semantic-search-elser-ingest-pipelines.md) for a complete example on adding documents to a `sparse_vector` mapped field using ELSER.
 
 ## Parameters for `sparse_vector` fields [sparse-vectors-params]
@@ -36,6 +63,38 @@ The following parameters are accepted by `sparse_vector` fields:
     * Exclude the field from [_source](/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#source-filtering).
     * Use [synthetic `_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source).
 
+index_options {applies_to}`stack: preview 9.1`
+:   (Optional, object) You can set index options for your  `sparse_vector` field to determine if you should prune tokens, and the parameter configurations for the token pruning. If pruning options are not set in your [`sparse_vector` query](/reference/query-languages/query-dsl/query-dsl-sparse-vector-query.md), Elasticsearch will use the default options configured for the field, if any.
+
+Parameters for `index_options` are:
+
+`prune` {applies_to}`stack: preview 9.1`
+:   (Optional, boolean) Whether to perform pruning, omitting the non-significant tokens from the query to improve query performance. If `prune` is true but the `pruning_config` is not specified, pruning will occur but default values will be used. Default: true.
+
+`pruning_config` {applies_to}`stack: preview 9.1`
+:   (Optional, object) Optional pruning configuration. If enabled, this will omit non-significant tokens from the query in order to improve query performance. This is only used if `prune` is set to `true`. If `prune` is set to `true` but `pruning_config` is not specified, default values will be used. If `prune` is set to false but `pruning_config` is specified, an exception will occur.
+
+    Parameters for `pruning_config` include:
+
+    `tokens_freq_ratio_threshold` {applies_to}`stack: preview 9.1`
+    :   (Optional, integer) Tokens whose frequency is more than `tokens_freq_ratio_threshold` times the average frequency of all tokens in the specified field are considered outliers and pruned. This value must between 1 and 100. Default: `5`.
+
+    `tokens_weight_threshold` {applies_to}`stack: preview 9.1`
+    :   (Optional, float) Tokens whose weight is less than `tokens_weight_threshold` are considered insignificant and pruned. This value must be between 0 and 1. Default: `0.4`.
+
+    ::::{note}
+    The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSERv2 that provided the most optimal results.
+    ::::
+
+When token pruning is applied, non-significant tokens will be pruned from the query.
+Non-significant tokens can be defined as tokens that meet both of the following criteria:
+* The token appears much more frequently than most tokens, indicating that it is a very common word and may not benefit the overall search results much.
+* The weight/score is so low that the token is likely not very relevant to the original term
+
+Both the token frequency threshold and weight threshold must show the token is non-significant in order for the token to be pruned.
+This ensures that:
+* The tokens that are kept are frequent enough and have significant scoring.
+* Very infrequent tokens that may not have as high of a score are removed.
 
 
 ## Multi-value sparse vectors [index-multi-value-sparse-vectors]

docs/reference/query-languages/esql/_snippets/functions/layout/kql.md

@@ -1,5 +1,5 @@
-* [preview] [`KQL`](../../functions-operators/search-functions.md#esql-kql)
-* [preview] [`MATCH`](../../functions-operators/search-functions.md#esql-match)
-* [preview] [`MATCH_PHRASE`](../../functions-operators/search-functions.md#esql-match_phrase)
-* [preview] [`QSTR`](../../functions-operators/search-functions.md#esql-qstr)
+* [`KQL`](../../functions-operators/search-functions.md#esql-kql)
+* [`MATCH`](../../functions-operators/search-functions.md#esql-match)
+* [`MATCH_PHRASE`](../../functions-operators/search-functions.md#esql-match_phrase)
+* [`QSTR`](../../functions-operators/search-functions.md#esql-qstr)
 % * [preview] [`TERM`](../../functions-operators/search-functions.md#esql-term)

docs/reference/query-languages/esql/_snippets/functions/layout/match.md

@@ -1,7 +1,6 @@
 ### Match operator (`:`) [esql-match-operator]
 ```{applies_to}
-stack: preview 9.0.0
-serverless: preview
+stack: preview 9.0, ga 9.1
 ```
 
 The only search operator is match (`:`).
@@ -13,12 +12,6 @@ The only search operator is match (`:`).
 :class: text-center
 :::
 
-
-::::{warning}
-Do not use on production environments. This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
-::::
-
-
 The match operator performs a [match query](/reference/query-languages/query-dsl/query-dsl-match-query.md) on the specified field. Returns true if the provided query matches the row.
 
 The match operator is equivalent to the [match function](../../../functions-operators/search-functions.md#esql-match).