From 5c09fbf8fe10cb26fd0f3c285b2ede8dd999725c Mon Sep 17 00:00:00 2001 From: kosabogi Date: Fri, 10 Oct 2025 13:42:02 +0200 Subject: [PATCH 1/4] Adds DiskBBQ to the BBQ documentation --- .../elasticsearch/index-settings/bbq.md | 88 +++++++++++++++++++ 1 file changed, 88 insertions(+) diff --git a/docs/reference/elasticsearch/index-settings/bbq.md b/docs/reference/elasticsearch/index-settings/bbq.md index a53cd6e9c83c6..5b5caa3dcd8b5 100644 --- a/docs/reference/elasticsearch/index-settings/bbq.md +++ b/docs/reference/elasticsearch/index-settings/bbq.md @@ -131,6 +131,94 @@ PUT bbq_flat-index } ``` +### `bbq_disk` [bbq-disk] + +```{applies_to} +stack: ga 9.2 +``` + +When you set a dense vector field’s `index_options` parameter to `type: bbq_disk`, {{es}} uses the DiskBBQ algorithm for efficient [kNN search](https://www.elastic.co/docs//solutions/search/vector/knn) on compressed vectors. DiskBBQ is an alternative to HNSW that stores most of the vector data on disk rather than in memory. It reduces memory usage and works well with large datasets or in low-memory environments. + +Instead of keeping the entire index in memory, DiskBBQ groups similar vectors into clusters on disk and searches only within the most relevant clusters. During a query, it identifies which clusters are closest to the query vector and compares only the vectors within those clusters. This approach reduces memory requirements while maintaining strong search relevance and speed. + +DiskBBQ typically achieves up to 95% recall. For use cases that require higher recall (99%+), consider using [bbq_hnsw](#bbq-hnsw) instead. + +The following example creates an index with a `dense_vector` field configured to use the `bbq_disk` algorithm. + +```console +PUT bbq_disk-index +{ + "mappings": { + "properties": { + "my_vector": { + "type": "dense_vector", + "dims": 3, + "similarity": "l2_norm", + "index": true, + "index_options": { + "type": "bbq_disk" + } + } + } + } +} +``` + +To change an existing index to use `bbq_disk`, update the field mapping: + +```console +PUT bbq_disk-index/_mapping +{ + "properties": { + "my_vector": { + "type": "dense_vector", + "dims": 64, + "index": true, + "index_options": { + "type": "bbq_disk" + } + } + } +} +``` + +To apply `bbq_disk` to all vectors at once, reindex them into a new index where the `index_options` parameter's `type` is set to `bbq_disk`: + +:::::{stepper} +::::{step} Create a destination index +```console +PUT my-index-bbq +{ + "mappings": { + "properties": { + "my_vector": { + "type": "dense_vector", + "dims": 64, + "index": true, + "index_options": { + "type": "bbq_disk" + } + } + } + } +} +``` +:::: + +::::{step} Reindex the data +```console +POST _reindex +{ + "source": { "index": "my-index" }, <1> + "dest": { "index": "my-index-bbq" } +} +``` +1. The existing index to be reindexed into the newly created index with the `bbq_disk` algorithm. +:::: + +::::: + + ## Oversampling [bbq-oversampling] Oversampling is a technique used with BBQ searches to reduce the accuracy loss from compression. Compression lowers the memory footprint by over 95% and improves query latency, at the cost of decreased result accuracy. This decrease can be mitigated by oversampling during query time and reranking the top results using the full vector. From ebb9dafb017b9eea87b0c3669eb374dd41fedb39 Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Mon, 13 Oct 2025 10:11:45 +0200 Subject: [PATCH 2/4] Update docs/reference/elasticsearch/index-settings/bbq.md Co-authored-by: Benjamin Trent --- docs/reference/elasticsearch/index-settings/bbq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/elasticsearch/index-settings/bbq.md b/docs/reference/elasticsearch/index-settings/bbq.md index 5b5caa3dcd8b5..2b79557a532e3 100644 --- a/docs/reference/elasticsearch/index-settings/bbq.md +++ b/docs/reference/elasticsearch/index-settings/bbq.md @@ -141,7 +141,7 @@ When you set a dense vector field’s `index_options` parameter to `type: bbq_di Instead of keeping the entire index in memory, DiskBBQ groups similar vectors into clusters on disk and searches only within the most relevant clusters. During a query, it identifies which clusters are closest to the query vector and compares only the vectors within those clusters. This approach reduces memory requirements while maintaining strong search relevance and speed. -DiskBBQ typically achieves up to 95% recall. For use cases that require higher recall (99%+), consider using [bbq_hnsw](#bbq-hnsw) instead. +DiskBBQ is well suited when the recall desired is 95%. For use cases that require exceptionally high recall (99%+) many vector clusters might have to be visited, negatively impacting performance. In very high recall cases, [bbq_hnsw](#bbq-hnsw), or one of the other `HNSW` formats might prove best depending on memory requirements. The following example creates an index with a `dense_vector` field configured to use the `bbq_disk` algorithm. From a0d03d3b16e8b5d46f84e53f52f2f86c0b77e28e Mon Sep 17 00:00:00 2001 From: kosabogi Date: Mon, 13 Oct 2025 10:22:45 +0200 Subject: [PATCH 3/4] Changes wording --- docs/reference/elasticsearch/index-settings/bbq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/elasticsearch/index-settings/bbq.md b/docs/reference/elasticsearch/index-settings/bbq.md index 2b79557a532e3..a8b8c0918a752 100644 --- a/docs/reference/elasticsearch/index-settings/bbq.md +++ b/docs/reference/elasticsearch/index-settings/bbq.md @@ -141,7 +141,7 @@ When you set a dense vector field’s `index_options` parameter to `type: bbq_di Instead of keeping the entire index in memory, DiskBBQ groups similar vectors into clusters on disk and searches only within the most relevant clusters. During a query, it identifies which clusters are closest to the query vector and compares only the vectors within those clusters. This approach reduces memory requirements while maintaining strong search relevance and speed. -DiskBBQ is well suited when the recall desired is 95%. For use cases that require exceptionally high recall (99%+) many vector clusters might have to be visited, negatively impacting performance. In very high recall cases, [bbq_hnsw](#bbq-hnsw), or one of the other `HNSW` formats might prove best depending on memory requirements. +DiskBBQ performs well when the desired recall is around 95%. For use cases that require exceptionally high recall (99% or higher), many vector clusters may need to be visited, which can negatively impact performance. In very high recall cases, [bbq_hnsw](#bbq-hnsw), or one of the other `HNSW` formats might prove best depending on memory requirements. The following example creates an index with a `dense_vector` field configured to use the `bbq_disk` algorithm. From b14d10c0b34c58cc88678e1d8c499e05a461a6ff Mon Sep 17 00:00:00 2001 From: kosabogi Date: Tue, 14 Oct 2025 13:28:33 +0200 Subject: [PATCH 4/4] Addresses feedback --- .../elasticsearch/index-settings/bbq.md | 24 ++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/reference/elasticsearch/index-settings/bbq.md b/docs/reference/elasticsearch/index-settings/bbq.md index a8b8c0918a752..947b8d5916b1a 100644 --- a/docs/reference/elasticsearch/index-settings/bbq.md +++ b/docs/reference/elasticsearch/index-settings/bbq.md @@ -137,11 +137,11 @@ PUT bbq_flat-index stack: ga 9.2 ``` -When you set a dense vector field’s `index_options` parameter to `type: bbq_disk`, {{es}} uses the DiskBBQ algorithm for efficient [kNN search](https://www.elastic.co/docs//solutions/search/vector/knn) on compressed vectors. DiskBBQ is an alternative to HNSW that stores most of the vector data on disk rather than in memory. It reduces memory usage and works well with large datasets or in low-memory environments. +When you set a dense vector field’s `index_options` parameter to `type: bbq_disk`, {{es}} uses the DiskBBQ algorithm, a disk-based alternative to HNSW for [kNN search](https://www.elastic.co/docs//solutions/search/vector/knn) on compressed vectors. DiskBBQ stores the vector data on disk instead of in memory, lowering RAM requirements and reducing the overall cost of vector storage and search. -Instead of keeping the entire index in memory, DiskBBQ groups similar vectors into clusters on disk and searches only within the most relevant clusters. During a query, it identifies which clusters are closest to the query vector and compares only the vectors within those clusters. This approach reduces memory requirements while maintaining strong search relevance and speed. +DiskBBQ groups similar vectors into small clusters using [hierarchical K-means](https://www.elastic.co/search-labs/blog/k-means-for-vector-indices). When processing a query, it finds the centroids closest to the query vector and only compares the vectors within those clusters. This targeted approach reduces the number of in-memory operations, making it ideal for large-scale or memory-constrained environments. -DiskBBQ performs well when the desired recall is around 95%. For use cases that require exceptionally high recall (99% or higher), many vector clusters may need to be visited, which can negatively impact performance. In very high recall cases, [bbq_hnsw](#bbq-hnsw), or one of the other `HNSW` formats might prove best depending on memory requirements. +DiskBBQ typically performs well for recall levels up to around 95%. For use cases requiring exceptionally high recall (99% or higher), many vector clusters may need to be visited, which can negatively impact performance. In very high recall cases, [bbq_hnsw](#bbq-hnsw), or other HNSW-based formats deliver better performance depending on memory availability. The following example creates an index with a `dense_vector` field configured to use the `bbq_disk` algorithm. @@ -218,6 +218,23 @@ POST _reindex ::::: +You can set the `visit_percentage` parameter to define the fraction of vectors visited per shard during search. + +```console +POST bbq_disk-index/_search +{ + "query": { + "knn": { + "field": "my_vector", + "query_vector": [0.0127, 0.1230, 0.3929], + "k": 10, + "visit_percentage": 10.0 + } + } +} +``` + +A lower `visit_percentage` can further reduce memory use and speed up queries, while a higher value can improve recall. Learn more about [top-level parameters for knn](/reference/query-languages/query-dsl/query-dsl-knn-query.md#knn-query-top-level-parameters) queries. ## Oversampling [bbq-oversampling] @@ -246,5 +263,6 @@ You can change oversampling from the default 3× to another value. Refer to [Ove ## Learn more [bbq-learn-more] - [Better Binary Quantization (BBQ) in Lucene and {{es}}](https://www.elastic.co/search-labs/blog/better-binary-quantization-lucene-elasticsearch) - Learn how BBQ works, its benefits, and how it reduces memory usage while preserving search accuracy. +- [Introducing a new vector storage format: DiskBBQ](https://www.elastic.co/search-labs/blog/diskbbq-elasticsearch-introduction) - Learn how DiskBBQ improves vector search in low-memory environments and compares to HNSW in speed and cost-effectiveness. - [Dense vector field type](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector) - Find code examples for using `bbq_hnsw` `index_type`. - [kNN search](https://www.elastic.co/docs/solutions/search/vector/knn) - Learn about the search algorithm that BBQ works with. \ No newline at end of file