Adding linear retriever to support weighted sums of sub-retrievers (#120222)

Author: pmpailis

docs/changelog/120222.yaml

@@ -0,0 +1,5 @@
+pr: 120222
+summary: Adding linear retriever to support weighted sums of sub-retrievers
+area: "Search"
+type: enhancement
+issues: []

docs/reference/rest-api/common-parms.asciidoc

@@ -1338,7 +1338,7 @@ that lower ranked documents have more influence. This value must be greater than
 equal to `1`. Defaults to `60`.
 end::rrf-rank-constant[]
 
-tag::rrf-rank-window-size[]
+tag::compound-retriever-rank-window-size[]
 `rank_window_size`::
 (Optional, integer)
 +
@@ -1347,15 +1347,54 @@ query. A higher value will improve result relevance at the cost of performance.
 ranked result set is pruned down to the search request's <<search-size-param, size>>.
 `rank_window_size` must be greater than or equal to `size` and greater than or equal to `1`.
 Defaults to the `size` parameter.
-end::rrf-rank-window-size[]
+end::compound-retriever-rank-window-size[]
 
-tag::rrf-filter[]
+tag::compound-retriever-filter[]
 `filter`::
 (Optional, <<query-dsl, query object or list of query objects>>)
 +
 Applies the specified <<query-dsl-bool-query, boolean query filter>> to all of the specified sub-retrievers,
 according to each retriever's specifications.
-end::rrf-filter[]
+end::compound-retriever-filter[]
+
+tag::linear-retriever-components[]
+`retrievers`::
+(Required, array of objects)
++
+A list of the sub-retrievers' configuration, that we will take into account and whose result sets
+we will merge through a weighted sum. Each configuration can have a different weight and normalization depending
+on the specified retriever.
+
+Each entry specifies the following parameters:
+
+* `retriever`::
+(Required, a <<retriever, retriever>> object)
++
+Specifies the retriever for which we will compute the top documents for. The retriever will produce `rank_window_size`
+results, which will later be merged based on the specified `weight` and `normalizer`.
+
+* `weight`::
+(Optional, float)
++
+The weight that each score of this retriever's top docs will be multiplied with. Must be greater or equal to 0. Defaults to 1.0.
+
+* `normalizer`::
+(Optional, String)
++
+Specifies how we will normalize the retriever's scores, before applying the specified `weight`.
+Available values are: `minmax`, and `none`. Defaults to `none`.
+
+** `none`
+** `minmax` :
+A `MinMaxScoreNormalizer` that normalizes scores based on the following formula
++
+```
+score = (score - min) / (max - min)
+```
+
+See also <<retrievers-examples-linear-retriever, this hybrid search example>> using a linear retriever on how to
+independently configure and apply normalizers to retrievers.
+end::linear-retriever-components[]
 
 tag::knn-rescore-vector[]
 

docs/reference/search/retriever.asciidoc

@@ -28,6 +28,9 @@ A <<standard-retriever, retriever>> that replaces the functionality of a traditi
 `knn`::
 A <<knn-retriever, retriever>> that replaces the functionality of a <<search-api-knn, knn search>>.
 
+`linear`::
+A <<linear-retriever, retriever>> that linearly combines the scores of other retrievers for the top documents.
+
 `rescorer`::
 A <<rescorer-retriever, retriever>> that replaces the functionality of the <<rescore, query rescorer>>.
 
@@ -45,6 +48,8 @@ A <<rule-retriever, retriever>> that applies contextual <<query-rules>> to pin o
 
 A standard retriever returns top documents from a traditional <<query-dsl, query>>.
 
+[discrete]
+[[standard-retriever-parameters]]
 ===== Parameters:
 
 `query`::
@@ -195,6 +200,8 @@ Documents matching these conditions will have increased relevancy scores.
 
 A kNN retriever returns top documents from a <<knn-search, k-nearest neighbor search (kNN)>>.
 
+[discrete]
+[[knn-retriever-parameters]]
 ===== Parameters
 
 `field`::
@@ -265,21 +272,37 @@ GET /restaurants/_search
 This value must be fewer than or equal to `num_candidates`.
 <5> The size of the initial candidate set from which the final `k` nearest neighbors are selected.
 
+[[linear-retriever]]
+==== Linear Retriever
+A retriever that normalizes and linearly combines the scores of other retrievers.
+
+[discrete]
+[[linear-retriever-parameters]]
+===== Parameters
+
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=linear-retriever-components]
+
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=compound-retriever-rank-window-size]
+
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=compound-retriever-filter]
+
 [[rrf-retriever]]
 ==== RRF Retriever
 
 An <<rrf, RRF>> retriever returns top documents based on the RRF formula, equally weighting two or more child retrievers.
 Reciprocal rank fusion (RRF) is a method for combining multiple result sets with different relevance indicators into a single result set.
 
+[discrete]
+[[rrf-retriever-parameters]]
 ===== Parameters
 
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-retrievers]
 
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-constant]
 
-include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-window-size]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=compound-retriever-rank-window-size]
 
-include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-filter]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=compound-retriever-filter]
 
 [discrete]
 [[rrf-retriever-example-hybrid]]
@@ -540,6 +563,8 @@ score = ln(score), if score < 0
 ----
 ====
 
+[discrete]
+[[text-similarity-reranker-retriever-parameters]]
 ===== Parameters
 
 `retriever`::

docs/reference/search/rrf.asciidoc

@@ -45,7 +45,7 @@ include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-retrievers]
 
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-constant]
 
-include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-window-size]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=compound-retriever-rank-window-size]
 
 An example request using RRF:
 
@@ -791,11 +791,11 @@ A more specific example of highlighting in RRF can also be found in the <<retrie
 
 ==== Inner hits in RRF
 
-The `rrf` retriever supports <<inner-hits,inner hits>> functionality, allowing you to retrieve 
-related nested or parent/child documents alongside your main search results. Inner hits can be 
-specified as part of any nested sub-retriever and will be propagated to the top-level parent 
-retriever. Note that the inner hit computation will take place only at end of `rrf` retriever's 
-evaluation on the top matching documents, and not as part of the query execution of the nested 
+The `rrf` retriever supports <<inner-hits,inner hits>> functionality, allowing you to retrieve
+related nested or parent/child documents alongside your main search results. Inner hits can be
+specified as part of any nested sub-retriever and will be propagated to the top-level parent
+retriever. Note that the inner hit computation will take place only at end of `rrf` retriever's
+evaluation on the top matching documents, and not as part of the query execution of the nested
 sub-retrievers.
 
 [IMPORTANT]