Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 3 additions & 14 deletions docs/reference/elasticsearch/mapping-reference/dense-vector.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ $$$dense-vector-similarity$$$
`l2_norm`
: Computes similarity based on the L2 distance (also known as Euclidean distance) between the vectors. The document `_score` is computed as `1 / (1 + l2_norm(query, vector)^2)`.

For `bit` vectors, instead of using `l2_norm`, the `hamming` distance between the vectors is used. The `_score` transformation is `(numBits - hamming(a, b)) / numBits`
For `bit` vectors, instead of using `l2_norm`, the `hamming` distance between the vectors is used. The `_score` transformation is `(numBits - hamming(a, b)) / numBits`

`dot_product`
: Computes the dot product of two unit vectors. This option provides an optimized way to perform cosine similarity. The constraints and computed score are defined by `element_type`.
Expand Down Expand Up @@ -267,14 +267,12 @@ $$$dense-vector-index-options$$$
: (Required, string) The type of kNN algorithm to use. Can be either any of:
* `hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) for scalable approximate kNN search. This supports all `element_type` values.
* `int8_hnsw` - The default index type for some float vectors:

* {applies_to}`stack: ga 9.1` Default for float vectors with less than 384 dimensions.
* {applies_to}`stack: ga 9.0` Default for float all vectors.

This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 4x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
* `int4_hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 8x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
* `bbq_hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically binary quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 32x at the cost of accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).

{applies_to}`stack: ga 9.1` `bbq_hnsw` is the default index type for float vectors with greater than or equal to 384 dimensions.
* `flat` - This utilizes a brute-force search algorithm for exact kNN search. This supports all `element_type` values.
* `int8_flat` - This utilizes a brute-force search algorithm in addition to automatically scalar quantization. Only supports `element_type` of `float`.
Expand All @@ -290,7 +288,6 @@ $$$dense-vector-index-options$$$
`confidence_interval`
: (Optional, float) Only applicable to `int8_hnsw`, `int4_hnsw`, `int8_flat`, and `int4_flat` index types. The confidence interval to use when quantizing the vectors. Can be any value between and including `0.90` and `1.0` or exactly `0`. When the value is `0`, this indicates that dynamic quantiles should be calculated for optimized quantization. When between `0.90` and `1.0`, this value restricts the values used when calculating the quantization thresholds. For example, a value of `0.95` will only use the middle 95% of the values when calculating the quantization thresholds (e.g. the highest and lowest 2.5% of values will be ignored). Defaults to `1/(dims + 1)` for `int8` quantized vectors and `0` for `int4` for dynamic quantile calculation.


`rescore_vector` {applies_to}`stack: preview 9.0, ga 9.1`
: (Optional, object) An optional section that configures automatic vector rescoring on knn queries for the given field. Only applicable to quantized index types.
:::::{dropdown} Properties of rescore_vector
Expand All @@ -304,14 +301,11 @@ $$$dense-vector-index-options$$$
:::::
::::



## Synthetic `_source` [dense-vector-synthetic-source]

`dense_vector` fields support [synthetic `_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source) .


## Indexing & Searching bit vectors [dense-vector-index-bit]
## Indexing and searching bit vectors [dense-vector-index-bit]

When using `element_type: bit`, this will treat all vectors as bit vectors. Bit vectors utilize only a single bit per dimension and are internally encoded as bytes. This can be useful for very high-dimensional vectors or models.

Expand Down Expand Up @@ -354,7 +348,6 @@ PUT my-bit-vectors

1. The number of dimensions that represents the number of bits


```console
POST /my-bit-vectors/_bulk?refresh
{"index": {"_id" : "1"}}
Expand All @@ -366,7 +359,6 @@ POST /my-bit-vectors/_bulk?refresh
1. 5 bytes representing the 40 bit dimensioned vector
2. A hexidecimal string representing the 40 bit dimensioned vector


Then, when searching, you can use the `knn` query to search for similar bit vectors:

```console
Expand Down Expand Up @@ -412,7 +404,6 @@ POST /my-bit-vectors/_search?filter_path=hits.hits
}
```


## Updatable field type [_updatable_field_type]

To better accommodate scaling and performance needs, updating the `type` setting in `index_options` is possible with the [Update Mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping), according to the following graph (jumps allowed):
Expand Down Expand Up @@ -477,5 +468,3 @@ Vectors indexed before this change will keep using the `flat` type (raw float32
In order to have all the vectors updated to the new type, either reindexing or force merging should be used.

For debugging purposes, it’s possible to inspect how many segments (and docs) exist for each `type` with the [Index Segments API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-segments).


Loading