Adding linear retriever to support weighted sums of sub-retrievers

docs/changelog/120222.yaml

@@ -0,0 +1,5 @@
+pr: 120222
+summary: Adding linear retriever to support weighted sums of sub-retrievers
+area: "Search"
+type: enhancement
+issues: []

docs/reference/rest-api/common-parms.asciidoc

@@ -1338,7 +1338,7 @@ that lower ranked documents have more influence. This value must be greater than
 equal to `1`. Defaults to `60`.
 end::rrf-rank-constant[]
 
-tag::rrf-rank-window-size[]
+tag::compound-retriever-rank-window-size[]
 `rank_window_size`::
 (Optional, integer)
 +
@@ -1347,12 +1347,63 @@ query. A higher value will improve result relevance at the cost of performance.
 ranked result set is pruned down to the search request's <<search-size-param, size>>.
 `rank_window_size` must be greater than or equal to `size` and greater than or equal to `1`.
 Defaults to the `size` parameter.
-end::rrf-rank-window-size[]
+end::compound-retriever-rank-window-size[]
 
-tag::rrf-filter[]
+tag::compound-retriever-filter[]
 `filter`::
 (Optional, <<query-dsl, query object or list of query objects>>)
 +
 Applies the specified <<query-dsl-bool-query, boolean query filter>> to all of the specified sub-retrievers,
 according to each retriever's specifications.
-end::rrf-filter[]
+end::compound-retriever-filter[]
+
+tag::linear-retriever-components[]
+`components`::
+(Required, array of `component` objects)
++
+A list of the components, i.e. the sub-retrievers' configuration, that we will take into account and whose result sets
+we will merge through a weighted sum. Each component can have a different weight and normalization depending
+on the specified retriever.
+
+Each `component` entry specifies the following parameters:
+
+* `retriever`::
+(Required, a <<retriever, retriever>> object)
++
+Specifies the retriever for which we will compute the top documents for. The retriever will produce `rank_window_size`
+results, which will later be merged based on the specified `weight` and `normalizer`.
+
+* `weight`::
+(Optional, float)
++
+The weight that each score of this retriever's top docs will be multiplied with. Defaults to 1.0.
+
+* `normalizer`::
+(Optional, String or Object)
++
+Specifies how we will normalize the retriever's scores, before applying the specified `weight`.
+We can either provide a string reference to use with the default values or further configure any normalizer
+using its specific properties. Available values are: `minmax`, and `none`. Defaults to `none`.
+
+** `none` : takes no argument
+** `minmax` :
+A `MinMaxScoreNormalizer` that normalizes scores based on the following formula
++
+```
+score = (score - min) / (max - min)
+```
+Available properties are:
+*** `min`::
+(Optional, float)
++
+The minimum value of the original scores. Defaults to result set's true min value.
+
+*** `max`::
+(Optional, float)
++
+The maximum value of the original scores. Defaults to result set's true max value.
+
+
+See also <<retrievers-examples-linear-retriever, this hybrid search example>> using a linear retriever on how to
+independently configure and apply normalizers to retrievers.
+end::linear-retriever-components[]

docs/reference/search/retriever.asciidoc

@@ -28,6 +28,9 @@ A <<standard-retriever, retriever>> that replaces the functionality of a traditi
 `knn`::
 A <<knn-retriever, retriever>> that replaces the functionality of a <<search-api-knn, knn search>>.
 
+`linear`::
+A <<linear-retriever, retriever>> that linearly combines the scores of other retrievers for the top documents.
+
 `rescorer`::
 A <<rescorer-retriever, retriever>> that replaces the functionality of the <<rescore, query rescorer>>.
 
@@ -263,6 +266,19 @@ GET /restaurants/_search
 This value must be fewer than or equal to `num_candidates`.
 <5> The size of the initial candidate set from which the final `k` nearest neighbors are selected.
 
+[[linear-retriever]]
+==== Linear Retriever
+A retriever that normalizes and linearly combines the scores of other retrievers. If the final scores produced after the
+weighted combination of all sub-retrievers are negative, they are set to increments of `1e-6` to avoid negative scores.
+
+===== Parameters
+
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=linear-retriever-components]
+
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=compound-retriever-rank-window-size]
+
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=compound-retriever-filter]
+
 [[rrf-retriever]]
 ==== RRF Retriever
 
@@ -275,9 +291,9 @@ include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-retrievers]
 
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-constant]
 
-include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-window-size]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=compound-retriever-rank-window-size]
 
-include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-filter]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=compound-retriever-filter]
 
 [discrete]
 [[rrf-retriever-example-hybrid]]
@@ -576,15 +592,15 @@ This example demonstrates how to deploy the {ml-docs}/ml-nlp-rerank.html[Elastic
 
 Follow these steps:
 
-. Create an inference endpoint for the `rerank` task using the <<put-inference-api, Create {infer} API>>. 
+. Create an inference endpoint for the `rerank` task using the <<put-inference-api, Create {infer} API>>.
 +
 [source,console]
 ----
 PUT _inference/rerank/my-elastic-rerank
 {
   "service": "elasticsearch",
   "service_settings": {
-    "model_id": ".rerank-v1", 
+    "model_id": ".rerank-v1",
     "num_threads": 1,
     "adaptive_allocations": { <1>
       "enabled": true,
@@ -595,7 +611,7 @@ PUT _inference/rerank/my-elastic-rerank
 }
 ----
 // TEST[skip:uses ML]
-<1> {ml-docs}/ml-nlp-auto-scale.html#nlp-model-adaptive-allocations[Adaptive allocations] will be enabled with the minimum of 1 and the maximum of 10 allocations. 
+<1> {ml-docs}/ml-nlp-auto-scale.html#nlp-model-adaptive-allocations[Adaptive allocations] will be enabled with the minimum of 1 and the maximum of 10 allocations.
 +
 . Define a `text_similarity_rerank` retriever:
 +

docs/reference/search/rrf.asciidoc

@@ -45,7 +45,7 @@ include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-retrievers]
 
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-constant]
 
-include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-window-size]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=compound-retriever-rank-window-size]
 
 An example request using RRF:
 
@@ -791,11 +791,11 @@ A more specific example of highlighting in RRF can also be found in the <<retrie
 
 ==== Inner hits in RRF
 
-The `rrf` retriever supports <<inner-hits,inner hits>> functionality, allowing you to retrieve 
-related nested or parent/child documents alongside your main search results. Inner hits can be 
-specified as part of any nested sub-retriever and will be propagated to the top-level parent 
-retriever. Note that the inner hit computation will take place only at end of `rrf` retriever's 
-evaluation on the top matching documents, and not as part of the query execution of the nested 
+The `rrf` retriever supports <<inner-hits,inner hits>> functionality, allowing you to retrieve
+related nested or parent/child documents alongside your main search results. Inner hits can be
+specified as part of any nested sub-retriever and will be propagated to the top-level parent
+retriever. Note that the inner hit computation will take place only at end of `rrf` retriever's
+evaluation on the top matching documents, and not as part of the query execution of the nested
 sub-retrievers.
 
 [IMPORTANT]