Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions docs/reference/how-to/knn-search.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -95,13 +95,20 @@ and https://elasticsearch-benchmarks.elastic.co/#tracks/dense_vector[here] some
of datasets and configurations that we use for our nightly benchmarks.

[discrete]
[[dense-vector-preloading]]
include::search-speed.asciidoc[tag=warm-fs-cache]

The following file extensions are used for the approximate kNN search:
Each extension is broken down by the quantization types.

* `vec` and `veq` for vector values
* `vex` for HNSW graph
* `vem`, `vemf`, and `vemq` for metadata
* `vex` for the HNSW graph
* `vec` for all non-quantized vector values. This includes all element types: `float`, `byte`, and `bit`.
* `veq` for quantized vectors indexed with <<dense-vector-quantization,`quantization`>>: `int4` or `int8`
* `veb` for binary vectors indexed with <<dense-vector-quantization,`quantization`>>: `bbq`
* `vem`, `vemf`, `vemq`, and `vemb` for metadata, usually small and not a concern for preloading
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for leaving vex out of this list, or was it just overlooked?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, it was indeed overlooked, let me add that.


Generally, if you are using a quantized index, you should only preload the relevant quantized values and the HNSW graph.
Preloading the raw vectors is not necessary and might be counterproductive.

[discrete]
=== Reduce the number of index segments
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/index-modules/store.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,8 @@ terms dictionaries, postings lists and points, which are the most important
parts of the index for search and aggregations.

For vector search, you use <<approximate-knn, approximate k-nearest neighbor search>>,
you might want to set the setting to vector search files: `["vec", "vex", "vem"]`
("vec" is used for vector values, "vex" – for HNSW graph, "vem" – for metadata).
you might want to set the setting to vector search files. See <<dense-vector-preloading, vector preloading>> for a detailed
list of the files.

Note that this setting can be dangerous on indices that are larger than the size
of the main memory of the host, as it would cause the filesystem cache to be
Expand Down