Release DiskBBQ(bbq_disk) index type for dense_vector fields (#135299)

This releases a new index type called DiskBBQ (configuration
`bbq_disk`). 

DiskBBQ is a cluster based format that provides:

 - faster and cheaper indexing than HNSW
 - Better behavior in lower memory environments (degrades linearly, not exponentially)
 - Is near HNSW for QPS when the index is in memory

Current restrictions that will be able to be removed in the future:

 - only floating point values are allowed currently
 - quantization is only to a single bit, so not recommended for low dimensionality vectors
 - all other restrictions for hnsw

To utilize the format, its just like any other:

```
PUT vectors {
  "mappings": {
    "properties": {
      "vector": {"type": "dense_vector", "index_options": {"type": "disk_bbq"}
    }
  }
}
```

Querying is just like any other field.

```
POST vectors/_search
{
    "query": {
        "knn": {
            "field": "vector",
            "query_vector": <vector,
            "k": 3
        }
    }
}
```

`num_candidates` can be used for tuning approximate nature of the
search. Or, more granular control can be provided by setting
`visit_percentage` directly.

Author: benwtrent

docs/changelog/135299.yaml

@@ -0,0 +1,47 @@
+pr: 135299
+summary: Release DiskBBQ(`bbq_disk`) index type for `dense_vector` fields
+area: Vector Search
+type: feature
+issues: []
+highlight:
+  title: Release DiskBBQ(`bbq_disk`) index type for `dense_vector` fields
+  body: |-
+     This provides a new index type called DiskBBQ (`bbq_disk`).
+     DiskBBQ is a cluster based format that provides:
+       - faster and cheaper indexing than HNSW
+       - Better behavior in lower memory environments (degrades linearly, not exponentially)
+       - Is near HNSW for QPS when the index is in memory
+
+     Current restrictions:
+       - only floating point values are allowed currently
+       - quantization is only to a single bit, so not recommended for low dimensionality vectors
+       - all other restrictions that exist for `dense_vector` fields still apply
+
+     To utilize the format, its just like any other:
+     [source,yaml]
+     ----------------------------
+     PUT vectors
+      {
+        "mappings": {
+          "properties": {
+            "vector": {"type": "dense_vector", "index_options": {"type": "disk_bbq"}
+          }
+        }
+      }
+     ----------------------------
+      Querying is just like any other field.
+     [source,yaml]
+     ----------------------------
+     POST vectors/_search{
+       "query": {
+         "knn": {
+           "field": "vector",
+           "query_vector": <vector>,
+           "k": 3
+         }
+       }
+     }
+     ----------------------------
+     `num_candidates` can be used for tuning approximate nature of the search.
+     Or, more granular control can be provided by setting `visit_percentage` directly.
+  notable: true

docs/reference/elasticsearch/mapping-reference/dense-vector.md

@@ -341,18 +341,19 @@ $$$dense-vector-index-options$$$
 `type`
 :   (Required, string) The type of kNN algorithm to use. Can be either any of:
     * `hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) for scalable approximate kNN search. This supports all `element_type` values.
-    * `int8_hnsw` - The default index type for some float vectors:      
+    * `int8_hnsw` - The default index type for some float vectors:
       * {applies_to}`stack: ga 9.1` Default for float vectors with less than 384 dimensions.
       * {applies_to}`stack: ga 9.0` Default for float all vectors.
       This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 4x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
     * `int4_hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 8x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
     * `bbq_hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically binary quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 32x at the cost of accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
-      
+
       {applies_to}`stack: ga 9.1` `bbq_hnsw` is the default index type for float vectors with greater than or equal to 384 dimensions.
     * `flat` - This utilizes a brute-force search algorithm for exact kNN search. This supports all `element_type` values.
-    * `int8_flat` - This utilizes a brute-force search algorithm in addition to automatically scalar quantization. Only supports `element_type` of `float`.
-    * `int4_flat` - This utilizes a brute-force search algorithm in addition to automatically half-byte scalar quantization. Only supports `element_type` of `float`.
-    * `bbq_flat` - This utilizes a brute-force search algorithm in addition to automatically binary quantization. Only supports `element_type` of `float`.
+    * `int8_flat` - This utilizes a brute-force search algorithm in addition to automatic scalar quantization. Only supports `element_type` of `float`.
+    * `int4_flat` - This utilizes a brute-force search algorithm in addition to automatic half-byte scalar quantization. Only supports `element_type` of `float`.
+    * `bbq_flat` - This utilizes a brute-force search algorithm in addition to automatic binary quantization. Only supports `element_type` of `float`.
+    * {applies_to}`stack: ga 9.2` `bbq_disk` - This utilizes a variant of [k-means clustering algorithm](https://en.wikipedia.org/wiki/K-means_clustering) in addition to automatic binary quantization to partition vectors and search subspaces rather than an entire graph structure as in with HNSW. Only supports `element_type` of `float`.  This combines the benefits of BBQ quantization with partitioning to further reduces the required memory overhead when compared with HNSW and can effectively be run at the smallest possible RAM and heap sizes when HNSW would otherwise cause swapping and grind to a halt.  DiskBBQ largely scales linearlly with the total RAM.  And search performance is enhanced at scale as a subset of the total vector space is loaded.
 
 `m`
 :   (Optional, integer) The number of neighbors each node will be connected to in the HNSW graph. Defaults to `16`. Only applicable to `hnsw`, `int8_hnsw`, `int4_hnsw` and `bbq_hnsw` index types.
@@ -363,6 +364,12 @@ $$$dense-vector-index-options$$$
 `confidence_interval`
 :   (Optional, float) Only applicable to `int8_hnsw`, `int4_hnsw`, `int8_flat`, and `int4_flat` index types. The confidence interval to use when quantizing the vectors. Can be any value between and including `0.90` and `1.0` or exactly `0`. When the value is `0`, this indicates that dynamic quantiles should be calculated for optimized quantization. When between `0.90` and `1.0`, this value restricts the values used when calculating the quantization thresholds. For example, a value of `0.95` will only use the middle 95% of the values when calculating the quantization thresholds (e.g. the highest and lowest 2.5% of values will be ignored). Defaults to `1/(dims + 1)` for `int8` quantized vectors and `0` for `int4` for dynamic quantile calculation.
 
+`default_visit_percentage` {applies_to}`stack: ga 9.2`
+:   (Optional, integer) Only applicable to `bbq_disk`. Must be between 0 and 100.  0 will default to using `num_candidates` for calculating the percent visited. Increasing `default_visit_percentage` tends to improve the accuracy of the final results. Defaults to ~1% per shard for every 1 million vectors.
+
+`cluster_size` {applies_to}`stack: ga 9.2`
+:   (Optional, integer) Only applicable to `bbq_disk`.  The number of vectors per cluster.  Smaller cluster sizes increases accuracy at the cost of performance. Defaults to `384`. Must be a value between `64` and `65536`.
+
 `rescore_vector` {applies_to}`stack: preview 9.0, ga 9.1`
 :   (Optional, object) An optional section that configures automatic vector rescoring on knn queries for the given field. Only applicable to quantized index types.
 :::::{dropdown} Properties of rescore_vector

docs/reference/elasticsearch/rest-apis/retrievers/knn-retriever.md

@@ -41,6 +41,12 @@ A kNN retriever returns top documents from a [k-nearest neighbor search (kNN)](d
     The number of nearest neighbor candidates to consider per shard. Needs to be greater than `k`, or `size` if `k` is omitted, and cannot exceed 10,000. {{es}} collects `num_candidates` results from each shard, then merges them to find the top `k` results. Increasing `num_candidates` tends to improve the accuracy of the final `k` results. Defaults to `Math.min(1.5 * k, 10_000)`.
 
 
+`visit_percentage` {applies_to}`stack: ga 9.2`
+:   (Optional, float)
+
+    The percentage of vectors to explore per shard while doing knn search with `bbq_disk`. Must be between 0 and 100.  0 will default to using `num_candidates` for calculating the percent visited. Increasing `visit_percentage` tends to improve the accuracy of the final results.  If `visit_percentage` is set for `bbq_disk`, `num_candidates` is ignored. Defaults to ~1% per shard for every 1 million vectors.
+
+
 `filter`
 :   (Optional, [query object or list of query objects](/reference/query-languages/querydsl.md))
 

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md

@@ -2,6 +2,9 @@
 navigation_title: "Knn"
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-knn-query.html
+applies_to:
+  stack: all
+  serverless: all
 ---
 
 # Knn query [query-dsl-knn-query]
@@ -87,6 +90,10 @@ If all queried fields are of type [semantic_text](/reference/elasticsearch/mappi
 :   (Optional, integer) The number of nearest neighbor candidates to consider per shard while doing knn search. Cannot exceed 10,000. Increasing `num_candidates` tends to improve the accuracy of the final results. Defaults to `1.5 * k` if `k` is set, or `1.5 * size` if `k` is not set.
 
 
+`visit_percentage` {applies_to}`stack: ga 9.2`
+:   (Optional, float) The percentage of vectors to explore per shard while doing knn search with `bbq_disk`. Must be between 0 and 100.  0 will default to using `num_candidates` for calculating the percent visited. Increasing `visit_percentage` tends to improve the accuracy of the final results.  If `visit_percentage` is set for `bbq_disk`, `num_candidates` is ignored. Defaults to ~1% per shard for every 1 million vectors.
+
+
 `filter`
 :   (Optional, query object) Query to filter the documents that can match. The kNN search will return the top documents that also match this filter. The value can be a single query or a list of queries. If `filter` is not provided, all documents are allowed to match.
 
@@ -108,15 +115,15 @@ The filter is a pre-filter, meaning that it is applied **during** the approximat
 :   (Optional, object) Apply oversampling and rescoring to quantized vectors.
 
     **Parameters for `rescore_vector`**:
-    
+
     `oversample`
     :   (Required, float)
 
         Applies the specified oversample factor to `k` on the approximate kNN search. The approximate kNN search will:
 
      * Retrieve `num_candidates` candidates per shard.
      * From these candidates, the top `k * oversample` candidates per shard will be rescored using the original vectors.
-     * The top `k` rescored candidates will be returned. Must be one of the following values: 
+     * The top `k` rescored candidates will be returned. Must be one of the following values:
        * \>= 1f to indicate the oversample factor
        * Exactly `0` to indicate that no oversampling and rescoring should occur. {applies_to}`stack: ga 9.1`
 

modules/percolator/src/internalClusterTest/java/org/elasticsearch/percolator/PercolatorQuerySearchIT.java

@@ -40,7 +40,6 @@
 import java.util.Collections;
 import java.util.Map;
 
-import static org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper.IVF_FORMAT;
 import static org.elasticsearch.index.query.QueryBuilders.boolQuery;
 import static org.elasticsearch.index.query.QueryBuilders.combinedFieldsQuery;
 import static org.elasticsearch.index.query.QueryBuilders.constantScoreQuery;
@@ -1360,15 +1359,7 @@ public void testKnnQueryNotSupportedInPercolator() throws IOException {
             """);
         indicesAdmin().prepareCreate("index1").setMapping(mappings).get();
         ensureGreen();
-        QueryBuilder knnVectorQueryBuilder = new KnnVectorQueryBuilder(
-            "my_vector",
-            new float[] { 1, 1, 1, 1, 1 },
-            10,
-            10,
-            IVF_FORMAT.isEnabled() ? 10f : null,
-            null,
-            null
-        );
+        QueryBuilder knnVectorQueryBuilder = new KnnVectorQueryBuilder("my_vector", new float[] { 1, 1, 1, 1, 1 }, 10, 10, 10f, null, null);
 
         IndexRequestBuilder indexRequestBuilder = prepareIndex("index1").setId("knn_query1")
             .setSource(jsonBuilder().startObject().field("my_query", knnVectorQueryBuilder).endObject());

qa/ccs-common-rest/src/yamlRestTest/java/org/elasticsearch/test/rest/yaml/CcsCommonYamlTestSuiteIT.java

@@ -99,7 +99,6 @@ public class CcsCommonYamlTestSuiteIT extends ESClientYamlSuiteTestCase {
         .setting("xpack.license.self_generated.type", "trial")
         .feature(FeatureFlag.TIME_SERIES_MODE)
         .feature(FeatureFlag.SUB_OBJECTS_AUTO_ENABLED)
-        .feature(FeatureFlag.IVF_FORMAT)
         .feature(FeatureFlag.SYNTHETIC_VECTORS);
 
     private static ElasticsearchCluster remoteCluster = ElasticsearchCluster.local()

qa/ccs-common-rest/src/yamlRestTest/java/org/elasticsearch/test/rest/yaml/RcsCcsCommonYamlTestSuiteIT.java

@@ -98,7 +98,6 @@ public class RcsCcsCommonYamlTestSuiteIT extends ESClientYamlSuiteTestCase {
         .setting("xpack.security.remote_cluster_client.ssl.enabled", "false")
         .feature(FeatureFlag.TIME_SERIES_MODE)
         .feature(FeatureFlag.SUB_OBJECTS_AUTO_ENABLED)
-        .feature(FeatureFlag.IVF_FORMAT)
         .feature(FeatureFlag.SYNTHETIC_VECTORS)
         .user("test_admin", "x-pack-test-password");
 

qa/smoke-test-multinode/src/yamlRestTest/java/org/elasticsearch/smoketest/SmokeTestMultiNodeClientYamlTestSuiteIT.java

@@ -37,7 +37,6 @@ public class SmokeTestMultiNodeClientYamlTestSuiteIT extends ESClientYamlSuiteTe
         .feature(FeatureFlag.TIME_SERIES_MODE)
         .feature(FeatureFlag.SUB_OBJECTS_AUTO_ENABLED)
         .feature(FeatureFlag.DOC_VALUES_SKIPPER)
-        .feature(FeatureFlag.IVF_FORMAT)
         .feature(FeatureFlag.SYNTHETIC_VECTORS)
         .build();
 

rest-api-spec/src/yamlRestTest/java/org/elasticsearch/test/rest/ClientYamlTestSuiteIT.java

@@ -37,7 +37,6 @@ public class ClientYamlTestSuiteIT extends ESClientYamlSuiteTestCase {
         .feature(FeatureFlag.TIME_SERIES_MODE)
         .feature(FeatureFlag.SUB_OBJECTS_AUTO_ENABLED)
         .feature(FeatureFlag.DOC_VALUES_SKIPPER)
-        .feature(FeatureFlag.IVF_FORMAT)
         .feature(FeatureFlag.SYNTHETIC_VECTORS)
         .build();
 

server/src/internalClusterTest/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldIndexTypeUpdateIT.java

@@ -24,7 +24,6 @@
 import org.elasticsearch.xcontent.XContentFactory;
 
 import java.io.IOException;
-import java.util.ArrayList;
 import java.util.Collection;
 import java.util.List;
 import java.util.Map;
@@ -51,12 +50,17 @@ public DenseVectorFieldIndexTypeUpdateIT(@Name("initialType") String initialType
 
     @ParametersFactory
     public static Collection<Object[]> params() {
-        List<String> types = new ArrayList<>(
-            List.of("flat", "int8_flat", "int4_flat", "bbq_flat", "hnsw", "int8_hnsw", "int4_hnsw", "bbq_hnsw")
+        List<String> types = List.of(
+            "flat",
+            "int8_flat",
+            "int4_flat",
+            "bbq_flat",
+            "hnsw",
+            "int8_hnsw",
+            "int4_hnsw",
+            "bbq_hnsw",
+            "bbq_disk"
         );
-        if (DenseVectorFieldMapper.IVF_FORMAT.isEnabled()) {
-            types.add("bbq_disk");
-        }
 
         // A type can be upgraded to types that follow in the list...
         List<Object[]> params = new java.util.ArrayList<>();