Skip to content

Commit 90db8f2

Browse files
kosabogibenwtrent
andauthored
[9.2] Adds DiskBBQ to the BBQ documentation (elastic#136371)
* Adds DiskBBQ to the BBQ documentation * Update docs/reference/elasticsearch/index-settings/bbq.md Co-authored-by: Benjamin Trent <[email protected]> * Changes wording * Addresses feedback --------- Co-authored-by: Benjamin Trent <[email protected]>
1 parent 7e1b7de commit 90db8f2

File tree

1 file changed

+106
-0
lines changed
  • docs/reference/elasticsearch/index-settings

1 file changed

+106
-0
lines changed

docs/reference/elasticsearch/index-settings/bbq.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,111 @@ PUT bbq_flat-index
131131
}
132132
```
133133

134+
### `bbq_disk` [bbq-disk]
135+
136+
```{applies_to}
137+
stack: ga 9.2
138+
```
139+
140+
When you set a dense vector field’s `index_options` parameter to `type: bbq_disk`, {{es}} uses the DiskBBQ algorithm, a disk-based alternative to HNSW for [kNN search](https://www.elastic.co/docs//solutions/search/vector/knn) on compressed vectors. DiskBBQ stores the vector data on disk instead of in memory, lowering RAM requirements and reducing the overall cost of vector storage and search.
141+
142+
DiskBBQ groups similar vectors into small clusters using [hierarchical K-means](https://www.elastic.co/search-labs/blog/k-means-for-vector-indices). When processing a query, it finds the centroids closest to the query vector and only compares the vectors within those clusters. This targeted approach reduces the number of in-memory operations, making it ideal for large-scale or memory-constrained environments.
143+
144+
DiskBBQ typically performs well for recall levels up to around 95%. For use cases requiring exceptionally high recall (99% or higher), many vector clusters may need to be visited, which can negatively impact performance. In very high recall cases, [bbq_hnsw](#bbq-hnsw), or other HNSW-based formats deliver better performance depending on memory availability.
145+
146+
The following example creates an index with a `dense_vector` field configured to use the `bbq_disk` algorithm.
147+
148+
```console
149+
PUT bbq_disk-index
150+
{
151+
"mappings": {
152+
"properties": {
153+
"my_vector": {
154+
"type": "dense_vector",
155+
"dims": 3,
156+
"similarity": "l2_norm",
157+
"index": true,
158+
"index_options": {
159+
"type": "bbq_disk"
160+
}
161+
}
162+
}
163+
}
164+
}
165+
```
166+
167+
To change an existing index to use `bbq_disk`, update the field mapping:
168+
169+
```console
170+
PUT bbq_disk-index/_mapping
171+
{
172+
"properties": {
173+
"my_vector": {
174+
"type": "dense_vector",
175+
"dims": 64,
176+
"index": true,
177+
"index_options": {
178+
"type": "bbq_disk"
179+
}
180+
}
181+
}
182+
}
183+
```
184+
185+
To apply `bbq_disk` to all vectors at once, reindex them into a new index where the `index_options` parameter's `type` is set to `bbq_disk`:
186+
187+
:::::{stepper}
188+
::::{step} Create a destination index
189+
```console
190+
PUT my-index-bbq
191+
{
192+
"mappings": {
193+
"properties": {
194+
"my_vector": {
195+
"type": "dense_vector",
196+
"dims": 64,
197+
"index": true,
198+
"index_options": {
199+
"type": "bbq_disk"
200+
}
201+
}
202+
}
203+
}
204+
}
205+
```
206+
::::
207+
208+
::::{step} Reindex the data
209+
```console
210+
POST _reindex
211+
{
212+
"source": { "index": "my-index" }, <1>
213+
"dest": { "index": "my-index-bbq" }
214+
}
215+
```
216+
1. The existing index to be reindexed into the newly created index with the `bbq_disk` algorithm.
217+
::::
218+
219+
:::::
220+
221+
You can set the `visit_percentage` parameter to define the fraction of vectors visited per shard during search.
222+
223+
```console
224+
POST bbq_disk-index/_search
225+
{
226+
"query": {
227+
"knn": {
228+
"field": "my_vector",
229+
"query_vector": [0.0127, 0.1230, 0.3929],
230+
"k": 10,
231+
"visit_percentage": 10.0
232+
}
233+
}
234+
}
235+
```
236+
237+
A lower `visit_percentage` can further reduce memory use and speed up queries, while a higher value can improve recall. Learn more about [top-level parameters for knn](/reference/query-languages/query-dsl/query-dsl-knn-query.md#knn-query-top-level-parameters) queries.
238+
134239
## Oversampling [bbq-oversampling]
135240

136241
Oversampling is a technique used with BBQ searches to reduce the accuracy loss from compression. Compression lowers the memory footprint by over 95% and improves query latency, at the cost of decreased result accuracy. This decrease can be mitigated by oversampling during query time and reranking the top results using the full vector.
@@ -158,5 +263,6 @@ You can change oversampling from the default 3× to another value. Refer to [Ove
158263
## Learn more [bbq-learn-more]
159264

160265
- [Better Binary Quantization (BBQ) in Lucene and {{es}}](https://www.elastic.co/search-labs/blog/better-binary-quantization-lucene-elasticsearch) - Learn how BBQ works, its benefits, and how it reduces memory usage while preserving search accuracy.
266+
- [Introducing a new vector storage format: DiskBBQ](https://www.elastic.co/search-labs/blog/diskbbq-elasticsearch-introduction) - Learn how DiskBBQ improves vector search in low-memory environments and compares to HNSW in speed and cost-effectiveness.
161267
- [Dense vector field type](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector) - Find code examples for using `bbq_hnsw` `index_type`.
162268
- [kNN search](https://www.elastic.co/docs/solutions/search/vector/knn) - Learn about the search algorithm that BBQ works with.

0 commit comments

Comments
 (0)