You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/elasticsearch/index-settings/bbq.md
+106Lines changed: 106 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -131,6 +131,111 @@ PUT bbq_flat-index
131
131
}
132
132
```
133
133
134
+
### `bbq_disk`[bbq-disk]
135
+
136
+
```{applies_to}
137
+
stack: ga 9.2
138
+
```
139
+
140
+
When you set a dense vector field’s `index_options` parameter to `type: bbq_disk`, {{es}} uses the DiskBBQ algorithm, a disk-based alternative to HNSW for [kNN search](https://www.elastic.co/docs//solutions/search/vector/knn) on compressed vectors. DiskBBQ stores the vector data on disk instead of in memory, lowering RAM requirements and reducing the overall cost of vector storage and search.
141
+
142
+
DiskBBQ groups similar vectors into small clusters using [hierarchical K-means](https://www.elastic.co/search-labs/blog/k-means-for-vector-indices). When processing a query, it finds the centroids closest to the query vector and only compares the vectors within those clusters. This targeted approach reduces the number of in-memory operations, making it ideal for large-scale or memory-constrained environments.
143
+
144
+
DiskBBQ typically performs well for recall levels up to around 95%. For use cases requiring exceptionally high recall (99% or higher), many vector clusters may need to be visited, which can negatively impact performance. In very high recall cases, [bbq_hnsw](#bbq-hnsw), or other HNSW-based formats deliver better performance depending on memory availability.
145
+
146
+
The following example creates an index with a `dense_vector` field configured to use the `bbq_disk` algorithm.
147
+
148
+
```console
149
+
PUT bbq_disk-index
150
+
{
151
+
"mappings": {
152
+
"properties": {
153
+
"my_vector": {
154
+
"type": "dense_vector",
155
+
"dims": 3,
156
+
"similarity": "l2_norm",
157
+
"index": true,
158
+
"index_options": {
159
+
"type": "bbq_disk"
160
+
}
161
+
}
162
+
}
163
+
}
164
+
}
165
+
```
166
+
167
+
To change an existing index to use `bbq_disk`, update the field mapping:
168
+
169
+
```console
170
+
PUT bbq_disk-index/_mapping
171
+
{
172
+
"properties": {
173
+
"my_vector": {
174
+
"type": "dense_vector",
175
+
"dims": 64,
176
+
"index": true,
177
+
"index_options": {
178
+
"type": "bbq_disk"
179
+
}
180
+
}
181
+
}
182
+
}
183
+
```
184
+
185
+
To apply `bbq_disk` to all vectors at once, reindex them into a new index where the `index_options` parameter's `type` is set to `bbq_disk`:
186
+
187
+
:::::{stepper}
188
+
::::{step} Create a destination index
189
+
```console
190
+
PUT my-index-bbq
191
+
{
192
+
"mappings": {
193
+
"properties": {
194
+
"my_vector": {
195
+
"type": "dense_vector",
196
+
"dims": 64,
197
+
"index": true,
198
+
"index_options": {
199
+
"type": "bbq_disk"
200
+
}
201
+
}
202
+
}
203
+
}
204
+
}
205
+
```
206
+
::::
207
+
208
+
::::{step} Reindex the data
209
+
```console
210
+
POST _reindex
211
+
{
212
+
"source": { "index": "my-index" }, <1>
213
+
"dest": { "index": "my-index-bbq" }
214
+
}
215
+
```
216
+
1. The existing index to be reindexed into the newly created index with the `bbq_disk` algorithm.
217
+
::::
218
+
219
+
:::::
220
+
221
+
You can set the `visit_percentage` parameter to define the fraction of vectors visited per shard during search.
222
+
223
+
```console
224
+
POST bbq_disk-index/_search
225
+
{
226
+
"query": {
227
+
"knn": {
228
+
"field": "my_vector",
229
+
"query_vector": [0.0127, 0.1230, 0.3929],
230
+
"k": 10,
231
+
"visit_percentage": 10.0
232
+
}
233
+
}
234
+
}
235
+
```
236
+
237
+
A lower `visit_percentage` can further reduce memory use and speed up queries, while a higher value can improve recall. Learn more about [top-level parameters for knn](/reference/query-languages/query-dsl/query-dsl-knn-query.md#knn-query-top-level-parameters) queries.
238
+
134
239
## Oversampling [bbq-oversampling]
135
240
136
241
Oversampling is a technique used with BBQ searches to reduce the accuracy loss from compression. Compression lowers the memory footprint by over 95% and improves query latency, at the cost of decreased result accuracy. This decrease can be mitigated by oversampling during query time and reranking the top results using the full vector.
@@ -158,5 +263,6 @@ You can change oversampling from the default 3× to another value. Refer to [Ove
158
263
## Learn more [bbq-learn-more]
159
264
160
265
-[Better Binary Quantization (BBQ) in Lucene and {{es}}](https://www.elastic.co/search-labs/blog/better-binary-quantization-lucene-elasticsearch) - Learn how BBQ works, its benefits, and how it reduces memory usage while preserving search accuracy.
266
+
-[Introducing a new vector storage format: DiskBBQ](https://www.elastic.co/search-labs/blog/diskbbq-elasticsearch-introduction) - Learn how DiskBBQ improves vector search in low-memory environments and compares to HNSW in speed and cost-effectiveness.
161
267
-[Dense vector field type](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector) - Find code examples for using `bbq_hnsw``index_type`.
162
268
-[kNN search](https://www.elastic.co/docs/solutions/search/vector/knn) - Learn about the search algorithm that BBQ works with.
0 commit comments