diff --git a/docs/reference/elasticsearch/index-settings/bbq.md b/docs/reference/elasticsearch/index-settings/bbq.md index a53cd6e9c83c6..947b8d5916b1a 100644 --- a/docs/reference/elasticsearch/index-settings/bbq.md +++ b/docs/reference/elasticsearch/index-settings/bbq.md @@ -131,6 +131,111 @@ PUT bbq_flat-index } ``` +### `bbq_disk` [bbq-disk] + +```{applies_to} +stack: ga 9.2 +``` + +When you set a dense vector field’s `index_options` parameter to `type: bbq_disk`, {{es}} uses the DiskBBQ algorithm, a disk-based alternative to HNSW for [kNN search](https://www.elastic.co/docs//solutions/search/vector/knn) on compressed vectors. DiskBBQ stores the vector data on disk instead of in memory, lowering RAM requirements and reducing the overall cost of vector storage and search. + +DiskBBQ groups similar vectors into small clusters using [hierarchical K-means](https://www.elastic.co/search-labs/blog/k-means-for-vector-indices). When processing a query, it finds the centroids closest to the query vector and only compares the vectors within those clusters. This targeted approach reduces the number of in-memory operations, making it ideal for large-scale or memory-constrained environments. + +DiskBBQ typically performs well for recall levels up to around 95%. For use cases requiring exceptionally high recall (99% or higher), many vector clusters may need to be visited, which can negatively impact performance. In very high recall cases, [bbq_hnsw](#bbq-hnsw), or other HNSW-based formats deliver better performance depending on memory availability. + +The following example creates an index with a `dense_vector` field configured to use the `bbq_disk` algorithm. + +```console +PUT bbq_disk-index +{ + "mappings": { + "properties": { + "my_vector": { + "type": "dense_vector", + "dims": 3, + "similarity": "l2_norm", + "index": true, + "index_options": { + "type": "bbq_disk" + } + } + } + } +} +``` + +To change an existing index to use `bbq_disk`, update the field mapping: + +```console +PUT bbq_disk-index/_mapping +{ + "properties": { + "my_vector": { + "type": "dense_vector", + "dims": 64, + "index": true, + "index_options": { + "type": "bbq_disk" + } + } + } +} +``` + +To apply `bbq_disk` to all vectors at once, reindex them into a new index where the `index_options` parameter's `type` is set to `bbq_disk`: + +:::::{stepper} +::::{step} Create a destination index +```console +PUT my-index-bbq +{ + "mappings": { + "properties": { + "my_vector": { + "type": "dense_vector", + "dims": 64, + "index": true, + "index_options": { + "type": "bbq_disk" + } + } + } + } +} +``` +:::: + +::::{step} Reindex the data +```console +POST _reindex +{ + "source": { "index": "my-index" }, <1> + "dest": { "index": "my-index-bbq" } +} +``` +1. The existing index to be reindexed into the newly created index with the `bbq_disk` algorithm. +:::: + +::::: + +You can set the `visit_percentage` parameter to define the fraction of vectors visited per shard during search. + +```console +POST bbq_disk-index/_search +{ + "query": { + "knn": { + "field": "my_vector", + "query_vector": [0.0127, 0.1230, 0.3929], + "k": 10, + "visit_percentage": 10.0 + } + } +} +``` + +A lower `visit_percentage` can further reduce memory use and speed up queries, while a higher value can improve recall. Learn more about [top-level parameters for knn](/reference/query-languages/query-dsl/query-dsl-knn-query.md#knn-query-top-level-parameters) queries. + ## Oversampling [bbq-oversampling] Oversampling is a technique used with BBQ searches to reduce the accuracy loss from compression. Compression lowers the memory footprint by over 95% and improves query latency, at the cost of decreased result accuracy. This decrease can be mitigated by oversampling during query time and reranking the top results using the full vector. @@ -158,5 +263,6 @@ You can change oversampling from the default 3× to another value. Refer to [Ove ## Learn more [bbq-learn-more] - [Better Binary Quantization (BBQ) in Lucene and {{es}}](https://www.elastic.co/search-labs/blog/better-binary-quantization-lucene-elasticsearch) - Learn how BBQ works, its benefits, and how it reduces memory usage while preserving search accuracy. +- [Introducing a new vector storage format: DiskBBQ](https://www.elastic.co/search-labs/blog/diskbbq-elasticsearch-introduction) - Learn how DiskBBQ improves vector search in low-memory environments and compares to HNSW in speed and cost-effectiveness. - [Dense vector field type](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector) - Find code examples for using `bbq_hnsw` `index_type`. - [kNN search](https://www.elastic.co/docs/solutions/search/vector/knn) - Learn about the search algorithm that BBQ works with. \ No newline at end of file