Skip to content

Commit ec97c05

Browse files
committed
Clarify the vector files utilized for preloading (#116488)
Adds clarification for vector preloading, what extension is to what storage kind, and that quantized vectors are stored in separate files allowing for individual preload. closes: #116273
1 parent 807d988 commit ec97c05

File tree

2 files changed

+12
-5
lines changed

2 files changed

+12
-5
lines changed

docs/reference/how-to/knn-search.asciidoc

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -95,13 +95,20 @@ and https://elasticsearch-benchmarks.elastic.co/#tracks/dense_vector[here] some
9595
of datasets and configurations that we use for our nightly benchmarks.
9696

9797
[discrete]
98+
[[dense-vector-preloading]]
9899
include::search-speed.asciidoc[tag=warm-fs-cache]
99100

100101
The following file extensions are used for the approximate kNN search:
102+
Each extension is broken down by the quantization types.
101103

102-
* `vec` and `veq` for vector values
103-
* `vex` for HNSW graph
104-
* `vem`, `vemf`, and `vemq` for metadata
104+
* `vex` for the HNSW graph
105+
* `vec` for all non-quantized vector values. This includes all element types: `float`, `byte`, and `bit`.
106+
* `veq` for quantized vectors indexed with <<dense-vector-quantization,`quantization`>>: `int4` or `int8`
107+
* `veb` for binary vectors indexed with <<dense-vector-quantization,`quantization`>>: `bbq`
108+
* `vem`, `vemf`, `vemq`, and `vemb` for metadata, usually small and not a concern for preloading
109+
110+
Generally, if you are using a quantized index, you should only preload the relevant quantized values and the HNSW graph.
111+
Preloading the raw vectors is not necessary and might be counterproductive.
105112

106113
[discrete]
107114
=== Reduce the number of index segments

docs/reference/index-modules/store.asciidoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -143,8 +143,8 @@ terms dictionaries, postings lists and points, which are the most important
143143
parts of the index for search and aggregations.
144144

145145
For vector search, you use <<approximate-knn, approximate k-nearest neighbor search>>,
146-
you might want to set the setting to vector search files: `["vec", "vex", "vem"]`
147-
("vec" is used for vector values, "vex" – for HNSW graph, "vem" – for metadata).
146+
you might want to set the setting to vector search files. See <<dense-vector-preloading, vector preloading>> for a detailed
147+
list of the files.
148148

149149
Note that this setting can be dangerous on indices that are larger than the size
150150
of the main memory of the host, as it would cause the filesystem cache to be

0 commit comments

Comments
 (0)