Merge branch 'main' into mp-index-template-registry

Author: nielsbauman

docs/changelog/127199.yaml

@@ -0,0 +1,6 @@
+pr: 127199
+summary: Disable a bugged commit
+area: ES|QL
+type: bug
+issues:
+ - 127197

docs/internal/Versioning.md

@@ -85,6 +85,25 @@ usually after each new minor release, to collapse the transport versions
 for the previous minor release. An example of such an operation can be found
 [here](https://github.com/elastic/elasticsearch/pull/104937).
 
+#### Tips
+
+- We collapse versions only on the `main` branch.
+- Use [TransportVersions.csv](../../server/src/main/resources/org/elasticsearch/TransportVersions.csv) as your guide.
+- For each release version listed in that file with a corresponding `INITIAL_ELASTICSEARCH_` entry corresponding to that version,
+  - change the prefix to `V_`
+  - replace all intervening entries since the previous patch `V_` entry with the new `V_` entry
+  - look at all uses of the new `V_` entry and look for dead code that can be deleted
+
+For example, if there's a version `1.2.3` in the CSV file,
+and `TransportVersions.java` has an entry called `INITIAL_ELASTICSEARCH_1_2_3`,
+then:
+- rename it to `V_1_2_3`,
+- replace any other intervening symbols between `V_1_2_2` and `V_1_2_3` with `V_1_2_3` itself, and
+- look through all the uses of the new `V_1_2_3` symbol; if any contain dead code, simplify it.
+
+When in doubt, you can always leave the code as-is.
+This is an optional cleanup step that is never required for correctness.
+
 ### Minimum compatibility versions
 
 The transport version used between two nodes is determined by the initial handshake

docs/reference/elasticsearch/mapping-reference/dense-vector.md

@@ -6,7 +6,6 @@ mapped_pages:
 
 # Dense vector field type [dense-vector]
 
-
 The `dense_vector` field type stores dense vectors of numeric values. Dense vector fields are primarily used for [k-nearest neighbor (kNN) search](docs-content://deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md).
 
 The `dense_vector` type does not support aggregations or sorting.
@@ -46,7 +45,6 @@ PUT my-index/_doc/2
 Unlike most other data types, dense vectors are always single-valued. It is not possible to store multiple values in one `dense_vector` field.
 ::::
 
-
 ## Index vectors for kNN search [index-vectors-knn-search]
 
 A *k-nearest neighbor* (kNN) search finds the *k* nearest vectors to a query vector, as measured by a similarity metric.
@@ -78,7 +76,6 @@ PUT my-index-2
 Indexing vectors for approximate kNN search is an expensive process. It can take substantial time to ingest documents that contain vector fields with `index` enabled. See [k-nearest neighbor (kNN) search](docs-content://deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md) to learn more about the memory requirements.
 ::::
 
-
 You can disable indexing by setting the `index` parameter to `false`:
 
 ```console
@@ -98,7 +95,6 @@ PUT my-index-2
 
 {{es}} uses the [HNSW algorithm](https://arxiv.org/abs/1603.09320) to support efficient kNN search. Like most kNN algorithms, HNSW is an approximate method that sacrifices result accuracy for improved speed.
 
-
 ## Automatically quantize vectors for kNN search [dense-vector-quantization]
 
 The `dense_vector` type supports quantization to reduce the memory footprint required when [searching](docs-content://solutions/search/vector/knn.md#approximate-knn) `float` vectors. The three following quantization strategies are supported:
@@ -117,17 +113,14 @@ Quantized vectors can use [oversampling and rescoring](docs-content://solutions/
 Quantization will continue to keep the raw float vector values on disk for reranking, reindexing, and quantization improvements over the lifetime of the data. This means disk usage will increase by ~25% for `int8`, ~12.5% for `int4`, and ~3.1% for `bbq` due to the overhead of storing the quantized and raw vectors.
 ::::
 
-
 ::::{note}
 `int4` quantization requires an even number of vector dimensions.
 ::::
 
-
 ::::{note}
 `bbq` quantization only supports vector dimensions that are greater than 64.
 ::::
 
-
 Here is an example of how to create a byte-quantized index:
 
 ```console
@@ -188,7 +181,6 @@ PUT my-byte-quantized-index
 }
 ```
 
-
 ## Parameters for dense vector fields [dense-vector-params]
 
 The following mapping parameters are accepted:
@@ -210,7 +202,6 @@ $$$dense-vector-element-type$$$
 
 ::::
 
-
 `dims`
 :   (Optional, integer) Number of vector dimensions. Can’t exceed `4096`. If `dims` is not specified, it will be set to the length of the first vector added to the field.
 
@@ -228,7 +219,6 @@ $$$dense-vector-similarity$$$
     `bit` vectors only support `l2_norm` as their similarity metric.
     ::::
 
-
 ::::{dropdown} Valid values for `similarity`
 `l2_norm`
 :   Computes similarity based on the L2 distance (also known as Euclidean distance) between the vectors. The document `_score` is computed as `1 / (1 + l2_norm(query, vector)^2)`.
@@ -242,7 +232,6 @@ For `bit` vectors, instead of using `l2_norm`, the `hamming` distance between th
 
     When `element_type` is `byte`, all vectors must have the same length including both document and query vectors or results will be inaccurate. The document `_score` is computed as `0.5 + (dot_product(query, vector) / (32768 * dims))` where `dims` is the number of dimensions per vector.
 
-
 `cosine`
 :   Computes the cosine similarity. During indexing {{es}} automatically normalizes vectors with `cosine` similarity to unit length. This allows to internally use `dot_product` for computing similarity, which is more efficient. Original un-normalized vectors can be still accessed through scripts. The document `_score` is computed as `(1 + cosine(query, vector)) / 2`. The `cosine` similarity does not allow vectors with zero magnitude, since cosine is not defined in this case.
 
@@ -251,12 +240,10 @@ For `bit` vectors, instead of using `l2_norm`, the `hamming` distance between th
 
 ::::
 
-
 ::::{note}
 Although they are conceptually related, the `similarity` parameter is different from [`text`](/reference/elasticsearch/mapping-reference/text.md) field [`similarity`](/reference/elasticsearch/mapping-reference/similarity.md) and accepts a distinct set of options.
 ::::
 
-
 $$$dense-vector-index-options$$$
 
 `index_options`
@@ -267,7 +254,6 @@ $$$dense-vector-index-options$$$
     ::::{dropdown} Properties of `index_options`
     `type`
     :   (Required, string) The type of kNN algorithm to use. Can be either any of:
-
         * `hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) for scalable approximate kNN search. This supports all `element_type` values.
         * `int8_hnsw` - The default index type for float vectors. This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 4x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
         * `int4_hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 8x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
@@ -276,17 +262,17 @@ $$$dense-vector-index-options$$$
         * `int8_flat` - This utilizes a brute-force search algorithm in addition to automatically scalar quantization. Only supports `element_type` of `float`.
         * `int4_flat` - This utilizes a brute-force search algorithm in addition to automatically half-byte scalar quantization. Only supports `element_type` of `float`.
         * `bbq_flat` - This utilizes a brute-force search algorithm in addition to automatically binary quantization. Only supports `element_type` of `float`.
-
-
+        
     `m`
     :   (Optional, integer) The number of neighbors each node will be connected to in the HNSW graph. Defaults to `16`. Only applicable to `hnsw`, `int8_hnsw`, `int4_hnsw` and `bbq_hnsw` index types.
-
+    
     `ef_construction`
     :   (Optional, integer) The number of candidates to track while assembling the list of nearest neighbors for each new node. Defaults to `100`. Only applicable to `hnsw`, `int8_hnsw`, `int4_hnsw` and `bbq_hnsw` index types.
-
+    
     `confidence_interval`
     :   (Optional, float) Only applicable to `int8_hnsw`, `int4_hnsw`, `int8_flat`, and `int4_flat` index types. The confidence interval to use when quantizing the vectors. Can be any value between and including `0.90` and `1.0` or exactly `0`. When the value is `0`, this indicates that dynamic quantiles should be calculated for optimized quantization. When between `0.90` and `1.0`, this value restricts the values used when calculating the quantization thresholds. For example, a value of `0.95` will only use the middle 95% of the values when calculating the quantization thresholds (e.g. the highest and lowest 2.5% of values will be ignored). Defaults to `1/(dims + 1)` for `int8` quantized vectors and `0` for `int4` for dynamic quantile calculation.
 
+
     `rescore_vector`
     :   (Optional, object) An optional section that configures automatic vector rescoring on knn queries for the given field. Only applicable to quantized index types.
     :::::{dropdown} Properties of `rescore_vector`

docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md

@@ -11,6 +11,8 @@ SLA of official GA features.
 index, to your {{esql}} query results, simplifying data enrichment
 and analysis workflows.
 
+Refer to [the high-level landing page](../../../../esql/esql-lookup-join.md) for an overview of the `LOOKUP JOIN` command, including use cases, prerequisites, and current limitations.
+
 **Syntax**
 
 ```esql
@@ -21,18 +23,14 @@ FROM <source_index>
 **Parameters**
 
 `<lookup_index>`
-: The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster
-  references are not supported.
+:   The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster references are not supported. Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).
 
 `<field_name>`
-: The field to join on. This field must exist
-  in both your current query results and in the lookup index. If the field
-  contains multi-valued entries, those entries will not match anything
-  (the added fields will contain `null` for those rows).
+:   The field to join on. This field must exist in both your current query results and in the lookup index. If the field contains multi-valued entries, those entries will not match anything (the added fields will contain `null` for those rows).
 
 **Description**
 
-The `LOOKUP JOIN` command adds new columns to your {esql} query
+The `LOOKUP JOIN` command adds new columns to your {{esql}} query
 results table by finding documents in a lookup index that share the same
 join field value as your result rows.
 

docs/reference/query-languages/esql/esql-lookup-join.md

@@ -16,7 +16,9 @@ For example, you can use `LOOKUP JOIN` to:
 * Quickly see if any source IPs match known malicious addresses.
 * Tag logs with the owning team or escalation info for faster triage and incident response.
 
-[`LOOKUP join`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) is similar to [`ENRICH`](/reference/query-languages/esql/commands/processing-commands.md#esql-enrich) in the fact that they both help you join data together. You should use `LOOKUP JOIN` when:
+## Compare with `ENRICH`
+
+[`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) is similar to [`ENRICH`](/reference/query-languages/esql/commands/processing-commands.md#esql-enrich) in the fact that they both help you join data together. You should use `LOOKUP JOIN` when:
 
 * Your enrichment data changes frequently
 * You want to avoid index-time processing
@@ -26,82 +28,119 @@ For example, you can use `LOOKUP JOIN` to:
 * You want to restrict users to use only specific lookup indices
 * You do not need to match using ranges or spatial relations
 
-## How the `LOOKUP JOIN` command works [esql-how-lookup-join-works]
+## How the command works [esql-how-lookup-join-works]
+
+The `LOOKUP JOIN` command adds fields from the lookup index as new columns to your results table based on matching values in the join field.
+
+The command requires two parameters:
+- The name of the lookup index (which must have the `lookup` [`index.mode setting`](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting))
+- The name of the field to join on
 
-The `LOOKUP JOIN` command adds new columns to a table, with data from {{es}} indices.
+```esql
+LOOKUP JOIN <lookup_index> ON <field_name>
+```
 
 :::{image} ../images/esql-lookup-join.png
-:alt: esql lookup join
+:alt: Illustration of the `LOOKUP JOIN` command, where the input table is joined with a lookup index to create an enriched output table.
 :::
 
-`<lookup_index>`
-: The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster references are not supported. Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).
-
-`<field_name>`
-: The field to join on. This field must exist in both your current query results and in the lookup index. If the field contains multi-valued entries, those entries will not match anything (the added fields will contain `null` for those rows).
+If you're familiar with SQL, `LOOKUP JOIN` has left-join behavior. This means that if no rows match in the lookup index, the incoming row is retained and `null`s are added. If many rows in the lookup index match, `LOOKUP JOIN` adds one row per match.
 
 ## Example
 
-`LOOKUP JOIN` has left-join behavior. If no rows match in the lookup index, `LOOKUP JOIN` retains the incoming row and adds `null`s. If many rows in the lookup index match, `LOOKUP JOIN` adds one row per match.
-
-In this example, we have two sample tables:
+You can run this example for yourself if you'd like to see how it works, by setting up the indices and adding sample data.
+
+### Sample data
+:::{dropdown} Expand for setup instructions
+
+**Set up indices**
+
+First let's create two indices with mappings: `threat_list` and `firewall_logs`.
+
+```console
+PUT threat_list
+{
+  "settings": {
+    "index.mode": "lookup" # The lookup index must use this mode
+  },
+  "mappings": {
+    "properties": {
+      "source.ip": { "type": "ip" },
+      "threat_level": { "type": "keyword" },
+      "threat_type": { "type": "keyword" },
+      "last_updated": { "type": "date" }
+    }
+  }
+}
+```
+```console
+PUT firewall_logs
+{
+  "mappings": {
+    "properties": {
+      "timestamp": { "type": "date" },
+      "source.ip": { "type": "ip" },
+      "destination.ip": { "type": "ip" },
+      "action": { "type": "keyword" },
+      "bytes_transferred": { "type": "long" }
+    }
+  }
+}
+```
 
-**employees**
+**Add sample data**
 
-| birth_date|emp_no|first_name|gender|hire_date|language|
-|---|---|---|---|---|---|
-|1955-10-04T00:00:00Z|10091|Amabile    |M|1992-11-18T00:00:00Z|3|
-|1964-10-18T00:00:00Z|10092|Valdiodio  |F|1989-09-22T00:00:00Z|1|
-|1964-06-11T00:00:00Z|10093|Sailaja    |M|1996-11-05T00:00:00Z|3|
-|1957-05-25T00:00:00Z|10094|Arumugam   |F|1987-04-18T00:00:00Z|5|
-|1965-01-03T00:00:00Z|10095|Hilari     |M|1986-07-15T00:00:00Z|4|
+Next, let's add some sample data to both indices. The `threat_list` index contains known malicious IPs, while the `firewall_logs` index contains logs of network traffic.
 
-**languages_non_unique_key**
+```console
+POST threat_list/_bulk
+{"index":{}}
+{"source.ip":"203.0.113.5","threat_level":"high","threat_type":"C2_SERVER","last_updated":"2025-04-22"}
+{"index":{}}
+{"source.ip":"198.51.100.2","threat_level":"medium","threat_type":"SCANNER","last_updated":"2025-04-23"}
+```
 
-|language_code|language_name|country|
-|---|---|---|
-|1|English|Canada|
-|1|English|
-|1||United Kingdom|
-|1|English|United States of America|
-|2|German|[Germany\|Austria]|
-|2|German|Switzerland|
-|2|German|
-|4|Spanish|
-|5||France|
-|[6\|7]|Mv-Lang|Mv-Land|
-|[7\|8]|Mv-Lang2|Mv-Land2|
-||Null-Lang|Null-Land|
-||Null-Lang2|Null-Land2|
+```console
+POST firewall_logs/_bulk
+{"index":{}}
+{"timestamp":"2025-04-23T10:00:01Z","source.ip":"192.0.2.1","destination.ip":"10.0.0.100","action":"allow","bytes_transferred":1024}
+{"index":{}}
+{"timestamp":"2025-04-23T10:00:05Z","source.ip":"203.0.113.5","destination.ip":"10.0.0.55","action":"allow","bytes_transferred":2048}
+{"index":{}}
+{"timestamp":"2025-04-23T10:00:08Z","source.ip":"198.51.100.2","destination.ip":"10.0.0.200","action":"block","bytes_transferred":0}
+{"index":{}}
+{"timestamp":"2025-04-23T10:00:15Z","source.ip":"203.0.113.5","destination.ip":"10.0.0.44","action":"allow","bytes_transferred":4096}
+{"index":{}}
+{"timestamp":"2025-04-23T10:00:30Z","source.ip":"192.0.2.1","destination.ip":"10.0.0.100","action":"allow","bytes_transferred":512}
+```
+:::
 
-Running the following query would provide the results shown below.
+### Query the data
 
 ```esql
-FROM employees
-| EVAL language_code = emp_no % 10
-| LOOKUP JOIN languages_lookup_non_unique_key ON language_code
-| WHERE emp_no > 10090 AND emp_no < 10096
-| SORT emp_no, country
-| KEEP emp_no, language_code, language_name, country;
+FROM firewall_logs # The source index
+| LOOKUP JOIN threat_list ON source.ip # The lookup index and join field
+| WHERE threat_level IS NOT NULL # Filter for rows non-null threat levels
+| SORT timestamp # LOOKUP JOIN does not guarantee output order, so you must explicitly sort the results if needed
+| KEEP timestamp, source.ip, destination.ip, action, threat_level, threat_type # Keep only relevant fields
+| LIMIT 10 # Limit the output to 10 rows
 ```
 
-|emp_no|language_code|language_name|country|
-|---|---|---|---|
-|    10091      | 1                     | English               | Canada|
-|    10091      | 1                     | null                  | United Kingdom|
-|    10091      | 1                     | English               | United States of America|
-|    10091      | 1                     | English               | null|
-|    10092      | 2                     | German                | [Germany, Austria]|
-|    10092      | 2                     | German                | Switzerland|
-|    10092      | 2                     | German                | null|
-|    10093      | 3                     | null                  | null|
-|    10094      | 4                     | Spanish               | null|
-|    10095      | 5                     | null                  | France|
-
-::::{important}
-`LOOKUP JOIN` does not guarantee the output to be in any particular order. If a certain order is required, users should use a [`SORT`](/reference/query-languages/esql/commands/processing-commands.md#esql-sort) somewhere after the `LOOKUP JOIN`.
-
-::::
+### Response
+
+A successful query will output a table. In this example, you can see that the `source.ip` field from the `firewall_logs` index is matched with the `source.ip` field in the `threat_list` index, and the corresponding `threat_level` and `threat_type` fields are added to the output.
+
+```
+   source.ip   |    action     |  threat_type  | threat_level  
+---------------+---------------+---------------+---------------
+203.0.113.5    |allow          |C2_SERVER      |high           
+198.51.100.2   |block          |SCANNER        |medium         
+203.0.113.5    |allow          |C2_SERVER      |high        
+```
+
+### Additional examples
+
+Refer to the examples section of the [`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) command reference for more examples.
 
 ## Prerequisites [esql-lookup-join-prereqs]