WIP - Add more docs

Author: carlosdelest

docs/reference/mapping/types/dense-vector.asciidoc

@@ -127,7 +127,7 @@ When using a quantized format, you may want to oversample and rescore the result
 To use a quantized index, you can set your index type to `int8_hnsw`, `int4_hnsw`, or `bbq_hnsw`. When indexing `float` vectors, the current default
 index type is `int8_hnsw`.
 
-Quantized vectors can use <<knn-quantized-vector-rescoring,rescoring>> to improve accuracy on approximate kNN search results.
+Quantized vectors can use <<dense-vector-knn-search-reranking,oversampling and rescoring>> to improve accuracy on approximate kNN search results.
 
 NOTE: Quantization will continue to keep the raw float vector values on disk for reranking, reindexing, and quantization improvements over the lifetime of the data.
 This means disk usage will increase by ~25% for `int8`, ~12.5% for `int4`, and ~3.1% for `bbq` due to the overhead of storing the quantized and raw vectors.

docs/reference/rest-api/common-parms.asciidoc

@@ -1360,5 +1360,5 @@ The approximate kNN search will retrieve the top `k * oversample` candidates per
 and then use the original vectors for rescoring.
 The top `k` rescored candidates will be returned as results.
 
-See <<knn-quantized-vector-rescoring,rescoring quantized vectors>> for details.
+See <<dense-vector-knn-search-reranking,oversampling and rescoring quantized vectors>> for details.
 end::knn-rescore[]

docs/reference/search/search-your-data/knn-search.asciidoc

@@ -781,7 +781,7 @@ What if you wanted to filter by some top-level document metadata? You can do thi
 
 
 NOTE: `filter` will always be over the top-level document metadata. This means you cannot filter based on `nested`
-      field metadata.
+field metadata.
 
 [source,console]
 ----
@@ -1012,55 +1012,6 @@ Now the result will contain the nearest found paragraph when searching.
 // TESTRESPONSE[s/"took": 4/"took" : "$body.took"/]
 
 
-[discrete]
-[[knn-quantized-vector-rescoring]]
-==== Rescoring results for quantized vectors
-
-When using <<dense-vector-quantization,quantized vectors>> for kNN search, you can optionally rescore results to balance performance and accuracy.
-Rescoring works by retrieving more results per shard using approximate kNN, and then use the original vector values for rescoring these results.
-As the non-quantized, original vectors are used to calculate the final score on the top results, rescoring combines:
-- The performance and memory gains of approximate retrieval using quantized vectors on the top candidates.
-- The accuracy of using the original vectors for rescoring the top candidates.
-
-Rescoring won't be as accurate as an <<exact-knn,exact kNN search>>, as some of the top results may not be retrieved using approximate kNN search.
-But the results retrieved by rescoring from the top candidates will have the same score and relative ordering as would be retrieved using exact kNN search.
-
-You can use the `rescore` option to specify an `oversample` parameter.
-When `oversample` is specified, the approximate kNN search will retrieve the top `k * oversample` candidates per shard.
-It will then use the original vectors to rescore them, and return the top `k` results.
-
-`num_candidates` will not be affected by oversample, besides ensuring that there are at least `k * oversample` candidates per shard.
-
-Here is an example of using the `rescore` option with the `oversample` parameter:
-
-[source,console]
-----
-POST image-index/_search
-{
-  "knn": {
-    "field": "image-vector",
-    "query_vector": [-5, 9, -12],
-    "k": 10,
-    "num_candidates": 100,
-    "rescore": {
-      "oversample": 2.0
-    }
-  },
-  "fields": [ "title", "file-type" ]
-}
-----
-//TEST[continued]
-// TEST[s/"k": 10/"k": 3/]
-// TEST[s/"num_candidates": 100/"num_candidates": 3/]
-
-This example will effectively:
-- Search using approximate kNN with `num_candidates` set to 100.
-- Rescore the top 20 (`k * oversample`) candidates per shard using the original vectors.
-- Return the top 10 (`k`) results from the rescored candidates.
-
-NOTE: Rescoring only makes sense for quantized vectors; when <<dense-vector-quantization,quantization>> is not used, the original vectors are used for scoring.
-Rescore option will be ignored for non-quantized `dense_vector` fields.
-
 [discrete]
 [[knn-indexing-considerations]]
 ==== Indexing considerations
@@ -1117,100 +1068,78 @@ NOTE: Approximate kNN search always uses the
 the global top `k` matches across shards. You cannot set the
 `search_type` explicitly when running kNN search.
 
+
 [discrete]
-[[exact-knn]]
-=== Exact kNN
+[[dense-vector-knn-search-reranking]]
+==== Oversampling and rescoring for quantized vectors
 
-To run an exact kNN search, use a `script_score` query with a vector function.
+When using <<dense-vector-quantization,quantized vectors>> for kNN search, you can optionally rescore results to balance performance and accuracy, by doing:
+* Oversampling: Retrieve more candidates per shard using approximate kNN
+* Rescoring: Use the original vector values for re-calculating the score on the oversampled candidates.
 
-. Explicitly map one or more `dense_vector` fields. If you don't intend to use
-the field for approximate kNN, set the `index` mapping option to `false`. This
-can significantly improve indexing speed.
-+
-[source,console]
-----
-PUT product-index
-{
-  "mappings": {
-    "properties": {
-      "product-vector": {
-        "type": "dense_vector",
-        "dims": 5,
-        "index": false
-      },
-      "price": {
-        "type": "long"
-      }
-    }
-  }
-}
-----
+As the non-quantized, original vectors are used to calculate the final score on the top results, rescoring combines:
 
-. Index your data.
-+
-[source,console]
-----
-POST product-index/_bulk?refresh=true
-{ "index": { "_id": "1" } }
-{ "product-vector": [230.0, 300.33, -34.8988, 15.555, -200.0], "price": 1599 }
-{ "index": { "_id": "2" } }
-{ "product-vector": [-0.5, 100.0, -13.0, 14.8, -156.0], "price": 799 }
-{ "index": { "_id": "3" } }
-{ "product-vector": [0.5, 111.3, -13.0, 14.8, -156.0], "price": 1099 }
-...
-----
-//TEST[continued]
-//TEST[s/\.\.\.//]
+* The performance and memory gains of approximate retrieval using quantized vectors on the top candidates.
+* The accuracy of using the original vectors for rescoring the top candidates.
+
+All forms of quantization will result in some accuracy loss and as the quantization level increases the accuracy loss will also increase.
+Generally, we have found that:
+
+* `int8` requires minimal if any rescoring
+* `int4` requires some rescoring for higher accuracy and larger recall scenarios. Generally, oversampling by 1.5x-2x recovers most of the accuracy loss.
+* `bbq` requires rescoring except on exceptionally large indices or models specifically designed for quantization. We have found that between 3x-5x oversampling is generally sufficient. But for fewer dimensions or vectors that do not quantize well, higher oversampling may be required.
+
+There are three main ways to oversample and rescore:
+
+* <<dense-vector-knn-search-reranking-rescore-parameter>>
+* <<dense-vector-knn-search-reranking-rescore-section>>
+* <<dense-vector-knn-search-reranking-script-score>>
+
+[discrete]
+[[dense-vector-knn-search-reranking-rescore-parameter]]
+===== Use the `rescore` option to rescore per shard
+
+preview:[]
+
+You can use the `rescore` option to automatically perform reranking.
+When a rescore `oversample` parameter is specified, the approximate kNN search will retrieve the top `k * oversample` candidates per shard.
+It will then use the original vectors to rescore them, and return the top `k` results.
+
+`num_candidates` will not be affected by oversample, besides ensuring that there are at least `k * oversample` candidates per shard.
+
+Here is an example of using the `rescore` option with the `oversample` parameter:
 
-. Use the <<search-search,search API>> to run a `script_score` query containing
-a <<vector-functions,vector function>>.
-+
-TIP: To limit the number of matched documents passed to the vector function, we
-recommend you specify a filter query in the `script_score.query` parameter. If
-needed, you can use a <<query-dsl-match-all-query,`match_all` query>> in this
-parameter to match all documents. However, matching all documents can
-significantly increase search latency.
-+
 [source,console]
 ----
-POST product-index/_search
+POST image-index/_search
 {
-  "query": {
-    "script_score": {
-      "query" : {
-        "bool" : {
-          "filter" : {
-            "range" : {
-              "price" : {
-                "gte": 1000
-              }
-            }
-          }
-        }
-      },
-      "script": {
-        "source": "cosineSimilarity(params.queryVector, 'product-vector') + 1.0",
-        "params": {
-          "queryVector": [-0.5, 90.0, -10, 14.8, -156.0]
-        }
-      }
+  "knn": {
+    "field": "image-vector",
+    "query_vector": [-5, 9, -12],
+    "k": 10,
+    "num_candidates": 100,
+    "rescore": {
+      "oversample": 2.0
     }
-  }
+  },
+  "fields": [ "title", "file-type" ]
 }
 ----
 //TEST[continued]
+// TEST[s/"k": 10/"k": 3/]
+// TEST[s/"num_candidates": 100/"num_candidates": 3/]
 
-[discrete]
-[[dense-vector-knn-search-reranking]]
-==== Oversampling and rescoring for quantized vectors
+This example will:
+* Search using approximate kNN with `num_candidates` set to 100.
+* Rescore the top 20 (`k * oversample`) candidates per shard using the original vectors.
+* Return the top 10 (`k`) results from the rescored candidates.
 
-All forms of quantization will result in some accuracy loss and as the quantization level increases the accuracy loss will also increase.
-Generally, we have found that:
-- `int8` requires minimal if any rescoring
-- `int4` requires some rescoring for higher accuracy and larger recall scenarios. Generally, oversampling by 1.5x-2x recovers most of the accuracy loss.
-- `bbq` requires rescoring except on exceptionally large indices or models specifically designed for quantization. We have found that between 3x-5x oversampling is generally sufficient. But for fewer dimensions or vectors that do not quantize well, higher oversampling may be required.
 
-There are two main ways to oversample and rescore. The first is to utilize the <<rescore, rescore section>> in the `_search` request.
+[discrete]
+[[dense-vector-knn-search-reranking-rescore-section]]
+===== Use the `rescore` section for top-level kNN search
+
+You can use the <<rescore, rescore section>> in the `_search` request to rescore the top results from a kNN search.
 
 Here is an example using the top level `knn` search with oversampling and using `rescore` to rerank the results:
 
@@ -1259,8 +1188,13 @@ gathering 20 nearest neighbors according to quantized scoring and rescoring with
 <5> The weight of the original query, here we simply throw away the original score
 <6> The weight of the rescore query, here we only use the rescore query
 
-The second way is to score per shard with the <<query-dsl-knn-query, knn query>> and <<query-dsl-script-score-query, script_score query >>. Generally, this means that there will be more rescoring per shard, but this
-can increase overall recall at the cost of compute.
+
+[discrete]
+[[dense-vector-knn-search-reranking-script-score]]
+===== Use a `script_score` query to rescore per shard
+
+You can rescore per shard with the <<query-dsl-knn-query, knn query>> and <<query-dsl-script-score-query, script_score query >>.
+Generally, this means that there will be more rescoring per shard, but this can increase overall recall at the cost of compute.
 
 [source,console]
 --------------------------------------------------
@@ -1292,3 +1226,87 @@ POST /my-index/_search
 <3> The number of candidates to use for the initial approximate `knn` search. This will search using the quantized vectors
 and return the top 20 candidates per shard to then be scored
 <4> The script to score the results. Script score will interact directly with the originally provided float32 vector.
+
+
+[discrete]
+[[exact-knn]]
+=== Exact kNN
+
+To run an exact kNN search, use a `script_score` query with a vector function.
+
+. Explicitly map one or more `dense_vector` fields. If you don't intend to use
+the field for approximate kNN, set the `index` mapping option to `false`. This
+can significantly improve indexing speed.
++
+[source,console]
+----
+PUT product-index
+{
+  "mappings": {
+    "properties": {
+      "product-vector": {
+        "type": "dense_vector",
+        "dims": 5,
+        "index": false
+      },
+      "price": {
+        "type": "long"
+      }
+    }
+  }
+}
+----
+
+. Index your data.
++
+[source,console]
+----
+POST product-index/_bulk?refresh=true
+{ "index": { "_id": "1" } }
+{ "product-vector": [230.0, 300.33, -34.8988, 15.555, -200.0], "price": 1599 }
+{ "index": { "_id": "2" } }
+{ "product-vector": [-0.5, 100.0, -13.0, 14.8, -156.0], "price": 799 }
+{ "index": { "_id": "3" } }
+{ "product-vector": [0.5, 111.3, -13.0, 14.8, -156.0], "price": 1099 }
+...
+----
+//TEST[continued]
+//TEST[s/\.\.\.//]
+
+. Use the <<search-search,search API>> to run a `script_score` query containing
+a <<vector-functions,vector function>>.
++
+TIP: To limit the number of matched documents passed to the vector function, we
+recommend you specify a filter query in the `script_score.query` parameter. If
+needed, you can use a <<query-dsl-match-all-query,`match_all` query>> in this
+parameter to match all documents. However, matching all documents can
+significantly increase search latency.
++
+[source,console]
+----
+POST product-index/_search
+{
+  "query": {
+    "script_score": {
+      "query" : {
+        "bool" : {
+          "filter" : {
+            "range" : {
+              "price" : {
+                "gte": 1000
+              }
+            }
+          }
+        }
+      },
+      "script": {
+        "source": "cosineSimilarity(params.queryVector, 'product-vector') + 1.0",
+        "params": {
+          "queryVector": [-0.5, 90.0, -10, 14.8, -156.0]
+        }
+      }
+    }
+  }
+}
+----
+//TEST[continued]