Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 106 additions & 0 deletions docs/reference/elasticsearch/index-settings/bbq.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,111 @@ PUT bbq_flat-index
}
```

### `bbq_disk` [bbq-disk]

```{applies_to}
stack: ga 9.2
```

When you set a dense vector field’s `index_options` parameter to `type: bbq_disk`, {{es}} uses the DiskBBQ algorithm, a disk-based alternative to HNSW for [kNN search](https://www.elastic.co/docs//solutions/search/vector/knn) on compressed vectors. DiskBBQ stores the vector data on disk instead of in memory, lowering RAM requirements and reducing the overall cost of vector storage and search.

DiskBBQ groups similar vectors into small clusters using [hierarchical K-means](https://www.elastic.co/search-labs/blog/k-means-for-vector-indices). When processing a query, it finds the centroids closest to the query vector and only compares the vectors within those clusters. This targeted approach reduces the number of in-memory operations, making it ideal for large-scale or memory-constrained environments.

DiskBBQ typically performs well for recall levels up to around 95%. For use cases requiring exceptionally high recall (99% or higher), many vector clusters may need to be visited, which can negatively impact performance. In very high recall cases, [bbq_hnsw](#bbq-hnsw), or other HNSW-based formats deliver better performance depending on memory availability.

The following example creates an index with a `dense_vector` field configured to use the `bbq_disk` algorithm.

```console
PUT bbq_disk-index
{
"mappings": {
"properties": {
"my_vector": {
"type": "dense_vector",
"dims": 3,
"similarity": "l2_norm",
"index": true,
"index_options": {
"type": "bbq_disk"
}
}
}
}
}
```

To change an existing index to use `bbq_disk`, update the field mapping:

```console
PUT bbq_disk-index/_mapping
{
"properties": {
"my_vector": {
"type": "dense_vector",
"dims": 64,
"index": true,
"index_options": {
"type": "bbq_disk"
}
}
}
}
```

To apply `bbq_disk` to all vectors at once, reindex them into a new index where the `index_options` parameter's `type` is set to `bbq_disk`:

:::::{stepper}
::::{step} Create a destination index
```console
PUT my-index-bbq
{
"mappings": {
"properties": {
"my_vector": {
"type": "dense_vector",
"dims": 64,
"index": true,
"index_options": {
"type": "bbq_disk"
}
}
}
}
}
```
::::

::::{step} Reindex the data
```console
POST _reindex
{
"source": { "index": "my-index" }, <1>
"dest": { "index": "my-index-bbq" }
}
```
1. The existing index to be reindexed into the newly created index with the `bbq_disk` algorithm.
::::

:::::

You can set the `visit_percentage` parameter to define the fraction of vectors visited per shard during search.

```console
POST bbq_disk-index/_search
{
"query": {
"knn": {
"field": "my_vector",
"query_vector": [0.0127, 0.1230, 0.3929],
"k": 10,
"visit_percentage": 10.0
}
}
}
```

A lower `visit_percentage` can further reduce memory use and speed up queries, while a higher value can improve recall. Learn more about [top-level parameters for knn](/reference/query-languages/query-dsl/query-dsl-knn-query.md#knn-query-top-level-parameters) queries.

## Oversampling [bbq-oversampling]

Oversampling is a technique used with BBQ searches to reduce the accuracy loss from compression. Compression lowers the memory footprint by over 95% and improves query latency, at the cost of decreased result accuracy. This decrease can be mitigated by oversampling during query time and reranking the top results using the full vector.
Expand Down Expand Up @@ -158,5 +263,6 @@ You can change oversampling from the default 3× to another value. Refer to [Ove
## Learn more [bbq-learn-more]

- [Better Binary Quantization (BBQ) in Lucene and {{es}}](https://www.elastic.co/search-labs/blog/better-binary-quantization-lucene-elasticsearch) - Learn how BBQ works, its benefits, and how it reduces memory usage while preserving search accuracy.
- [Introducing a new vector storage format: DiskBBQ](https://www.elastic.co/search-labs/blog/diskbbq-elasticsearch-introduction) - Learn how DiskBBQ improves vector search in low-memory environments and compares to HNSW in speed and cost-effectiveness.
- [Dense vector field type](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector) - Find code examples for using `bbq_hnsw` `index_type`.
- [kNN search](https://www.elastic.co/docs/solutions/search/vector/knn) - Learn about the search algorithm that BBQ works with.
Loading