You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/elasticsearch/mapping-reference/dense-vector.md
+4-18Lines changed: 4 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,6 @@ mapped_pages:
6
6
7
7
# Dense vector field type [dense-vector]
8
8
9
-
10
9
The `dense_vector` field type stores dense vectors of numeric values. Dense vector fields are primarily used for [k-nearest neighbor (kNN) search](docs-content://deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md).
11
10
12
11
The `dense_vector` type does not support aggregations or sorting.
@@ -46,7 +45,6 @@ PUT my-index/_doc/2
46
45
Unlike most other data types, dense vectors are always single-valued. It is not possible to store multiple values in one `dense_vector` field.
47
46
::::
48
47
49
-
50
48
## Index vectors for kNN search [index-vectors-knn-search]
51
49
52
50
A *k-nearest neighbor* (kNN) search finds the *k* nearest vectors to a query vector, as measured by a similarity metric.
@@ -78,7 +76,6 @@ PUT my-index-2
78
76
Indexing vectors for approximate kNN search is an expensive process. It can take substantial time to ingest documents that contain vector fields with `index` enabled. See [k-nearest neighbor (kNN) search](docs-content://deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md) to learn more about the memory requirements.
79
77
::::
80
78
81
-
82
79
You can disable indexing by setting the `index` parameter to `false`:
83
80
84
81
```console
@@ -98,7 +95,6 @@ PUT my-index-2
98
95
99
96
{{es}} uses the [HNSW algorithm](https://arxiv.org/abs/1603.09320) to support efficient kNN search. Like most kNN algorithms, HNSW is an approximate method that sacrifices result accuracy for improved speed.
100
97
101
-
102
98
## Automatically quantize vectors for kNN search [dense-vector-quantization]
103
99
104
100
The `dense_vector` type supports quantization to reduce the memory footprint required when [searching](docs-content://solutions/search/vector/knn.md#approximate-knn)`float` vectors. The three following quantization strategies are supported:
@@ -117,17 +113,14 @@ Quantized vectors can use [oversampling and rescoring](docs-content://solutions/
117
113
Quantization will continue to keep the raw float vector values on disk for reranking, reindexing, and quantization improvements over the lifetime of the data. This means disk usage will increase by ~25% for `int8`, ~12.5% for `int4`, and ~3.1% for `bbq` due to the overhead of storing the quantized and raw vectors.
118
114
::::
119
115
120
-
121
116
::::{note}
122
117
`int4` quantization requires an even number of vector dimensions.
123
118
::::
124
119
125
-
126
120
::::{note}
127
121
`bbq` quantization only supports vector dimensions that are greater than 64.
128
122
::::
129
123
130
-
131
124
Here is an example of how to create a byte-quantized index:
132
125
133
126
```console
@@ -188,7 +181,6 @@ PUT my-byte-quantized-index
188
181
}
189
182
```
190
183
191
-
192
184
## Parameters for dense vector fields [dense-vector-params]
: (Optional, integer) Number of vector dimensions. Can’t exceed `4096`. If `dims` is not specified, it will be set to the length of the first vector added to the field.
216
207
@@ -228,7 +219,6 @@ $$$dense-vector-similarity$$$
228
219
`bit` vectors only support `l2_norm` as their similarity metric.
229
220
::::
230
221
231
-
232
222
::::{dropdown} Valid values for `similarity`
233
223
`l2_norm`
234
224
: Computes similarity based on the L2 distance (also known as Euclidean distance) between the vectors. The document `_score` is computed as `1 / (1 + l2_norm(query, vector)^2)`.
@@ -242,7 +232,6 @@ For `bit` vectors, instead of using `l2_norm`, the `hamming` distance between th
242
232
243
233
When `element_type` is `byte`, all vectors must have the same length including both document and query vectors or results will be inaccurate. The document `_score` is computed as `0.5 + (dot_product(query, vector) / (32768 * dims))` where `dims` is the number of dimensions per vector.
244
234
245
-
246
235
`cosine`
247
236
: Computes the cosine similarity. During indexing {{es}} automatically normalizes vectors with `cosine` similarity to unit length. This allows to internally use `dot_product` for computing similarity, which is more efficient. Original un-normalized vectors can be still accessed through scripts. The document `_score` is computed as `(1 + cosine(query, vector)) / 2`. The `cosine` similarity does not allow vectors with zero magnitude, since cosine is not defined in this case.
248
237
@@ -251,12 +240,10 @@ For `bit` vectors, instead of using `l2_norm`, the `hamming` distance between th
251
240
252
241
::::
253
242
254
-
255
243
::::{note}
256
244
Although they are conceptually related, the `similarity` parameter is different from [`text`](/reference/elasticsearch/mapping-reference/text.md) field [`similarity`](/reference/elasticsearch/mapping-reference/similarity.md) and accepts a distinct set of options.
: (Required, string) The type of kNN algorithm to use. Can be either any of:
270
-
271
257
* `hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) for scalable approximate kNN search. This supports all `element_type` values.
272
258
* `int8_hnsw` - The default index type for float vectors. This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 4x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
273
259
* `int4_hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 8x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
* `int8_flat` - This utilizes a brute-force search algorithm in addition to automatically scalar quantization. Only supports `element_type` of `float`.
277
263
* `int4_flat` - This utilizes a brute-force search algorithm in addition to automatically half-byte scalar quantization. Only supports `element_type` of `float`.
278
264
* `bbq_flat` - This utilizes a brute-force search algorithm in addition to automatically binary quantization. Only supports `element_type` of `float`.
279
-
280
-
265
+
281
266
`m`
282
267
: (Optional, integer) The number of neighbors each node will be connected to in the HNSW graph. Defaults to `16`. Only applicable to `hnsw`, `int8_hnsw`, `int4_hnsw` and `bbq_hnsw` index types.
283
-
268
+
284
269
`ef_construction`
285
270
: (Optional, integer) The number of candidates to track while assembling the list of nearest neighbors for each new node. Defaults to `100`. Only applicable to `hnsw`, `int8_hnsw`, `int4_hnsw` and `bbq_hnsw` index types.
286
-
271
+
287
272
`confidence_interval`
288
273
: (Optional, float) Only applicable to `int8_hnsw`, `int4_hnsw`, `int8_flat`, and `int4_flat` index types. The confidence interval to use when quantizing the vectors. Can be any value between and including `0.90` and `1.0` or exactly `0`. When the value is `0`, this indicates that dynamic quantiles should be calculated for optimized quantization. When between `0.90` and `1.0`, this value restricts the values used when calculating the quantization thresholds. For example, a value of `0.95` will only use the middle 95% of the values when calculating the quantization thresholds (e.g. the highest and lowest 2.5% of values will be ignored). Defaults to `1/(dims + 1)` for `int8` quantized vectors and `0` for `int4` for dynamic quantile calculation.
289
274
275
+
290
276
`rescore_vector`
291
277
: (Optional, object) An optional section that configures automatic vector rescoring on knn queries for the given field. Only applicable to quantized index types.
Copy file name to clipboardExpand all lines: docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md
+5-7Lines changed: 5 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,6 +11,8 @@ SLA of official GA features.
11
11
index, to your {{esql}} query results, simplifying data enrichment
12
12
and analysis workflows.
13
13
14
+
Refer to [the high-level landing page](../../../../esql/esql-lookup-join.md) for an overview of the `LOOKUP JOIN` command, including use cases, prerequisites, and current limitations.
15
+
14
16
**Syntax**
15
17
16
18
```esql
@@ -21,18 +23,14 @@ FROM <source_index>
21
23
**Parameters**
22
24
23
25
`<lookup_index>`
24
-
: The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster
25
-
references are not supported.
26
+
: The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster references are not supported. Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).
26
27
27
28
`<field_name>`
28
-
: The field to join on. This field must exist
29
-
in both your current query results and in the lookup index. If the field
30
-
contains multi-valued entries, those entries will not match anything
31
-
(the added fields will contain `null` for those rows).
29
+
: The field to join on. This field must exist in both your current query results and in the lookup index. If the field contains multi-valued entries, those entries will not match anything (the added fields will contain `null` for those rows).
32
30
33
31
**Description**
34
32
35
-
The `LOOKUP JOIN` command adds new columns to your {esql} query
33
+
The `LOOKUP JOIN` command adds new columns to your {{esql}} query
36
34
results table by finding documents in a lookup index that share the same
Copy file name to clipboardExpand all lines: docs/reference/query-languages/esql/esql-lookup-join.md
+99-60Lines changed: 99 additions & 60 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,9 @@ For example, you can use `LOOKUP JOIN` to:
16
16
* Quickly see if any source IPs match known malicious addresses.
17
17
* Tag logs with the owning team or escalation info for faster triage and incident response.
18
18
19
-
[`LOOKUP join`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) is similar to [`ENRICH`](/reference/query-languages/esql/commands/processing-commands.md#esql-enrich) in the fact that they both help you join data together. You should use `LOOKUP JOIN` when:
19
+
## Compare with `ENRICH`
20
+
21
+
[`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) is similar to [`ENRICH`](/reference/query-languages/esql/commands/processing-commands.md#esql-enrich) in the fact that they both help you join data together. You should use `LOOKUP JOIN` when:
20
22
21
23
* Your enrichment data changes frequently
22
24
* You want to avoid index-time processing
@@ -26,82 +28,119 @@ For example, you can use `LOOKUP JOIN` to:
26
28
* You want to restrict users to use only specific lookup indices
27
29
* You do not need to match using ranges or spatial relations
28
30
29
-
## How the `LOOKUP JOIN` command works [esql-how-lookup-join-works]
31
+
## How the command works [esql-how-lookup-join-works]
32
+
33
+
The `LOOKUP JOIN` command adds fields from the lookup index as new columns to your results table based on matching values in the join field.
34
+
35
+
The command requires two parameters:
36
+
- The name of the lookup index (which must have the `lookup`[`index.mode setting`](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting))
37
+
- The name of the field to join on
30
38
31
-
The `LOOKUP JOIN` command adds new columns to a table, with data from {{es}} indices.
39
+
```esql
40
+
LOOKUP JOIN <lookup_index> ON <field_name>
41
+
```
32
42
33
43
:::{image} ../images/esql-lookup-join.png
34
-
:alt: esql lookup join
44
+
:alt: Illustration of the `LOOKUP JOIN` command, where the input table is joined with a lookup index to create an enriched output table.
35
45
:::
36
46
37
-
`<lookup_index>`
38
-
: The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster references are not supported. Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).
39
-
40
-
`<field_name>`
41
-
: The field to join on. This field must exist in both your current query results and in the lookup index. If the field contains multi-valued entries, those entries will not match anything (the added fields will contain `null` for those rows).
47
+
If you're familiar with SQL, `LOOKUP JOIN` has left-join behavior. This means that if no rows match in the lookup index, the incoming row is retained and `null`s are added. If many rows in the lookup index match, `LOOKUP JOIN` adds one row per match.
42
48
43
49
## Example
44
50
45
-
`LOOKUP JOIN` has left-join behavior. If no rows match in the lookup index, `LOOKUP JOIN` retains the incoming row and adds `null`s. If many rows in the lookup index match, `LOOKUP JOIN` adds one row per match.
46
-
47
-
In this example, we have two sample tables:
51
+
You can run this example for yourself if you'd like to see how it works, by setting up the indices and adding sample data.
52
+
53
+
### Sample data
54
+
:::{dropdown} Expand for setup instructions
55
+
56
+
**Set up indices**
57
+
58
+
First let's create two indices with mappings: `threat_list` and `firewall_logs`.
59
+
60
+
```console
61
+
PUT threat_list
62
+
{
63
+
"settings": {
64
+
"index.mode": "lookup" # The lookup index must use this mode
Next, let's add some sample data to both indices. The `threat_list` index contains known malicious IPs, while the `firewall_logs` index contains logs of network traffic.
`LOOKUP JOIN` does not guarantee the output to be in any particular order. If a certain order is required, users should use a [`SORT`](/reference/query-languages/esql/commands/processing-commands.md#esql-sort) somewhere after the `LOOKUP JOIN`.
103
-
104
-
::::
129
+
### Response
130
+
131
+
A successful query will output a table. In this example, you can see that the `source.ip` field from the `firewall_logs` index is matched with the `source.ip` field in the `threat_list` index, and the corresponding `threat_level` and `threat_type` fields are added to the output.
Refer to the examples section of the [`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) command reference for more examples.
0 commit comments