You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/elasticsearch/mapping-reference/dense-vector.md
+4-18Lines changed: 4 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,6 @@ mapped_pages:
6
6
7
7
# Dense vector field type [dense-vector]
8
8
9
-
10
9
The `dense_vector` field type stores dense vectors of numeric values. Dense vector fields are primarily used for [k-nearest neighbor (kNN) search](docs-content://deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md).
11
10
12
11
The `dense_vector` type does not support aggregations or sorting.
@@ -46,7 +45,6 @@ PUT my-index/_doc/2
46
45
Unlike most other data types, dense vectors are always single-valued. It is not possible to store multiple values in one `dense_vector` field.
47
46
::::
48
47
49
-
50
48
## Index vectors for kNN search [index-vectors-knn-search]
51
49
52
50
A *k-nearest neighbor* (kNN) search finds the *k* nearest vectors to a query vector, as measured by a similarity metric.
@@ -78,7 +76,6 @@ PUT my-index-2
78
76
Indexing vectors for approximate kNN search is an expensive process. It can take substantial time to ingest documents that contain vector fields with `index` enabled. See [k-nearest neighbor (kNN) search](docs-content://deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md) to learn more about the memory requirements.
79
77
::::
80
78
81
-
82
79
You can disable indexing by setting the `index` parameter to `false`:
83
80
84
81
```console
@@ -98,7 +95,6 @@ PUT my-index-2
98
95
99
96
{{es}} uses the [HNSW algorithm](https://arxiv.org/abs/1603.09320) to support efficient kNN search. Like most kNN algorithms, HNSW is an approximate method that sacrifices result accuracy for improved speed.
100
97
101
-
102
98
## Automatically quantize vectors for kNN search [dense-vector-quantization]
103
99
104
100
The `dense_vector` type supports quantization to reduce the memory footprint required when [searching](docs-content://solutions/search/vector/knn.md#approximate-knn)`float` vectors. The three following quantization strategies are supported:
@@ -117,17 +113,14 @@ Quantized vectors can use [oversampling and rescoring](docs-content://solutions/
117
113
Quantization will continue to keep the raw float vector values on disk for reranking, reindexing, and quantization improvements over the lifetime of the data. This means disk usage will increase by ~25% for `int8`, ~12.5% for `int4`, and ~3.1% for `bbq` due to the overhead of storing the quantized and raw vectors.
118
114
::::
119
115
120
-
121
116
::::{note}
122
117
`int4` quantization requires an even number of vector dimensions.
123
118
::::
124
119
125
-
126
120
::::{note}
127
121
`bbq` quantization only supports vector dimensions that are greater than 64.
128
122
::::
129
123
130
-
131
124
Here is an example of how to create a byte-quantized index:
132
125
133
126
```console
@@ -188,7 +181,6 @@ PUT my-byte-quantized-index
188
181
}
189
182
```
190
183
191
-
192
184
## Parameters for dense vector fields [dense-vector-params]
: (Optional, integer) Number of vector dimensions. Can’t exceed `4096`. If `dims` is not specified, it will be set to the length of the first vector added to the field.
216
207
@@ -228,7 +219,6 @@ $$$dense-vector-similarity$$$
228
219
`bit` vectors only support `l2_norm` as their similarity metric.
229
220
::::
230
221
231
-
232
222
::::{dropdown} Valid values for `similarity`
233
223
`l2_norm`
234
224
: Computes similarity based on the L2 distance (also known as Euclidean distance) between the vectors. The document `_score` is computed as `1 / (1 + l2_norm(query, vector)^2)`.
@@ -242,7 +232,6 @@ For `bit` vectors, instead of using `l2_norm`, the `hamming` distance between th
242
232
243
233
When `element_type` is `byte`, all vectors must have the same length including both document and query vectors or results will be inaccurate. The document `_score` is computed as `0.5 + (dot_product(query, vector) / (32768 * dims))` where `dims` is the number of dimensions per vector.
244
234
245
-
246
235
`cosine`
247
236
: Computes the cosine similarity. During indexing {{es}} automatically normalizes vectors with `cosine` similarity to unit length. This allows to internally use `dot_product` for computing similarity, which is more efficient. Original un-normalized vectors can be still accessed through scripts. The document `_score` is computed as `(1 + cosine(query, vector)) / 2`. The `cosine` similarity does not allow vectors with zero magnitude, since cosine is not defined in this case.
248
237
@@ -251,12 +240,10 @@ For `bit` vectors, instead of using `l2_norm`, the `hamming` distance between th
251
240
252
241
::::
253
242
254
-
255
243
::::{note}
256
244
Although they are conceptually related, the `similarity` parameter is different from [`text`](/reference/elasticsearch/mapping-reference/text.md) field [`similarity`](/reference/elasticsearch/mapping-reference/similarity.md) and accepts a distinct set of options.
: (Required, string) The type of kNN algorithm to use. Can be either any of:
270
-
271
257
* `hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) for scalable approximate kNN search. This supports all `element_type` values.
272
258
* `int8_hnsw` - The default index type for float vectors. This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 4x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
273
259
* `int4_hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 8x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
* `int8_flat` - This utilizes a brute-force search algorithm in addition to automatically scalar quantization. Only supports `element_type` of `float`.
277
263
* `int4_flat` - This utilizes a brute-force search algorithm in addition to automatically half-byte scalar quantization. Only supports `element_type` of `float`.
278
264
* `bbq_flat` - This utilizes a brute-force search algorithm in addition to automatically binary quantization. Only supports `element_type` of `float`.
279
-
280
-
265
+
281
266
`m`
282
267
: (Optional, integer) The number of neighbors each node will be connected to in the HNSW graph. Defaults to `16`. Only applicable to `hnsw`, `int8_hnsw`, `int4_hnsw` and `bbq_hnsw` index types.
283
-
268
+
284
269
`ef_construction`
285
270
: (Optional, integer) The number of candidates to track while assembling the list of nearest neighbors for each new node. Defaults to `100`. Only applicable to `hnsw`, `int8_hnsw`, `int4_hnsw` and `bbq_hnsw` index types.
286
-
271
+
287
272
`confidence_interval`
288
273
: (Optional, float) Only applicable to `int8_hnsw`, `int4_hnsw`, `int8_flat`, and `int4_flat` index types. The confidence interval to use when quantizing the vectors. Can be any value between and including `0.90` and `1.0` or exactly `0`. When the value is `0`, this indicates that dynamic quantiles should be calculated for optimized quantization. When between `0.90` and `1.0`, this value restricts the values used when calculating the quantization thresholds. For example, a value of `0.95` will only use the middle 95% of the values when calculating the quantization thresholds (e.g. the highest and lowest 2.5% of values will be ignored). Defaults to `1/(dims + 1)` for `int8` quantized vectors and `0` for `int4` for dynamic quantile calculation.
0 commit comments