Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions docs/reference/rest-api/common-parms.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -1326,13 +1326,21 @@ that lower ranked documents have more influence. This value must be greater than
equal to `1`. Defaults to `60`.
end::rrf-rank-constant[]

tag::rrf-window-size[]
`window_size`::
tag::rrf-rank-window-size[]
`rank_window_size`::
(Optional, integer)
+
This value determines the size of the individual result sets per
query. A higher value will improve result relevance at the cost of performance. The final
ranked result set is pruned down to the search request's <<search-size-param, size>>.
`window_size` must be greater than or equal to `size` and greater than or equal to `1`.
`rank_window_size` must be greater than or equal to `size` and greater than or equal to `1`.
Defaults to the `size` parameter.
end::rrf-window-size[]
end::rrf-rank-window-size[]

tag::rrf-filter[]
`filter`::
(Optional, <<query-dsl, query object or list of query objects>>)
+
Applies the specified <<query-dsl-bool-query, boolean query filter>> to all of the specified sub-retrievers,
according to each retriever's specifications.
end::rrf-filter[]
23 changes: 18 additions & 5 deletions docs/reference/search/retriever.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ GET my-embeddings/_search

An <<rrf, RRF>> retriever returns top documents based on the RRF formula,
equally weighting two or more child retrievers.
Reciprocal rank fusion (RRF) is a method for combining multiple result
Reciprocal rank fusion (RRF) is a method for combining multiple result
sets with different relevance indicators into a single result set.

===== Parameters
Expand All @@ -207,7 +207,9 @@ include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-retrievers]

include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-constant]

include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-window-size]
include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-window-size]

include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-filter]

===== Restrictions

Expand All @@ -225,7 +227,7 @@ A simple hybrid search example (lexical search + dense vector search) combining
----
GET /restaurants/_search
{
"retriever": {
"retriever": {
"rrf": { <1>
"retrievers": [ <2>
{
Expand Down Expand Up @@ -340,6 +342,10 @@ Currently you can:
** Refer to the <<text-similarity-reranker-retriever-example-eland,example>> on this page for a step-by-step guide.

===== Parameters
`retriever`::
(Required, <<retriever, retriever>>)
+
The child retriever that generates the initial set of top documents to be re-ranked.

`field`::
(Required, `string`)
Expand All @@ -366,6 +372,13 @@ The number of top documents to consider in the re-ranking process. Defaults to `
+
Sets a minimum threshold score for including documents in the re-ranked results. Documents with similarity scores below this threshold will be excluded. Note that score calculations vary depending on the model used.

`filter`::
(Optional, <<query-dsl, query object or list of query objects>>)
+
Applies the specified <<query-dsl-bool-query, boolean query filter>> to the child <<retriever, retriever>>.
If the child retriever already specifies any filters, then this top-level filter is applied in conjuction
with the filter defined in the child retriever.

===== Restrictions

A text similarity re-ranker retriever is a compound retriever. Child retrievers may not use elements that are restricted by having a compound retriever as part of the retriever tree.
Expand Down Expand Up @@ -441,13 +454,13 @@ eland_import_hub_model \
+
[source,js]
----
PUT _inference/rerank/my-msmarco-minilm-model
PUT _inference/rerank/my-msmarco-minilm-model
{
"service": "elasticsearch",
"service_settings": {
"num_allocations": 1,
"num_threads": 1,
"model_id": "cross-encoder__ms-marco-minilm-l-6-v2"
"model_id": "cross-encoder__ms-marco-minilm-l-6-v2"
}
}
----
Expand Down
6 changes: 2 additions & 4 deletions docs/reference/search/rrf.asciidoc
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
[[rrf]]
=== Reciprocal rank fusion

preview::["This functionality is in technical preview and may be changed or removed in a future release.
The syntax will likely change before GA.
Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features."]
preview::["This functionality is in technical preview and may be changed or removed in a future release. The syntax will likely change before GA. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features."]

https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf[Reciprocal rank fusion (RRF)]
is a method for combining multiple result sets with different relevance indicators into a single result set.
Expand Down Expand Up @@ -43,7 +41,7 @@ include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-retrievers]

include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-constant]

include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-window-size]
include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=rrf-rank-window-size]

An example request using RRF:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,23 @@ For implementation details, including notable restrictions, check out the

[discrete]
[[retrievers-overview-types]]
==== Retriever types
==== Retriever types

Retrievers come in various types, each tailored for different search operations.
The following retrievers are currently available:

* <<standard-retriever,*Standard Retriever*>>. Returns top documents from a
traditional https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl.html[query].
Mimics a traditional query but in the context of a retriever framework. This
ensures backward compatibility as existing `_search` requests remain supported.
That way you can transition to the new abstraction at your own pace without
* <<standard-retriever,*Standard Retriever*>>. Returns top documents from a
traditional https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl.html[query].
Mimics a traditional query but in the context of a retriever framework. This
ensures backward compatibility as existing `_search` requests remain supported.
That way you can transition to the new abstraction at your own pace without
mixing syntaxes.
* <<knn-retriever,*kNN Retriever*>>. Returns top documents from a <<search-api-knn,knn search>>,
* <<knn-retriever,*kNN Retriever*>>. Returns top documents from a <<search-api-knn,knn search>>,
in the context of a retriever framework.
* <<rrf-retriever,*RRF Retriever*>>. Combines and ranks multiple first-stage retrievers using
the reciprocal rank fusion (RRF) algorithm. Allows you to combine multiple result sets
the reciprocal rank fusion (RRF) algorithm. Allows you to combine multiple result sets
with different relevance indicators into a single result set.
An RRF retriever is a *compound retriever*, where its `filter` element is
An RRF retriever is a *compound retriever*, where its `filter` element is
propagated to its sub retrievers.
+
Sub retrievers may not use elements that are restricted by having a compound retriever as part of the retriever tree.
Expand All @@ -38,7 +38,7 @@ See the <<rrf-using-multiple-standard-retrievers,RRF documentation>> for detaile
Requires first creating a `rerank` task using the <<put-inference-api,{es} Inference API>>.

[discrete]
==== What makes retrievers useful?
==== What makes retrievers useful?

Here's an overview of what makes retrievers useful and how they differ from regular queries.

Expand Down Expand Up @@ -140,7 +140,7 @@ GET example-index/_search
],
"rank":{
"rrf":{
"window_size":50,
"rank_window_size":50,
"rank_constant":20
}
}
Expand All @@ -155,14 +155,14 @@ GET example-index/_search

Here are some important terms:

* *Retrieval Pipeline*. Defines the entire retrieval and ranking logic to
* *Retrieval Pipeline*. Defines the entire retrieval and ranking logic to
produce top hits.
* *Retriever Tree*. A hierarchical structure that defines how retrievers interact.
* *First-stage Retriever*. Returns an initial set of candidate documents.
* *Compound Retriever*. Builds on one or more retrievers,
* *Compound Retriever*. Builds on one or more retrievers,
enhancing document retrieval and ranking logic.
* *Combiners*. Compound retrievers that merge top hits
from multiple sub-retrievers.
* *Combiners*. Compound retrievers that merge top hits
from multiple sub-retrievers.
* *Rerankers*. Special compound retrievers that reorder hits and may adjust the number of hits, with distinctions between first-stage and second-stage rerankers.

[discrete]
Expand All @@ -180,4 +180,4 @@ Refer to the {kibana-ref}/playground.html[Playground documentation] for more inf
[[retrievers-overview-api-reference]]
==== API reference

For implementation details, including notable restrictions, check out the <<retriever,reference documentation>> in the Search API docs.
For implementation details, including notable restrictions, check out the <<retriever,reference documentation>> in the Search API docs.
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@

/**
* This search phase is responsible for executing any re-ranking needed for the given search request, iff that is applicable.
* It starts by retrieving {@code num_shards * window_size} results from the query phase and reduces them to a global list of
* the top {@code window_size} results. It then reaches out to the shards to extract the needed feature data,
* It starts by retrieving {@code num_shards * rank_window_size} results from the query phase and reduces them to a global list of
* the top {@code rank_window_size} results. It then reaches out to the shards to extract the needed feature data,
* and finally passes all this information to the appropriate {@code RankFeatureRankCoordinatorContext} which is responsible for reranking
* the results. If no rank query is specified, it proceeds directly to the next phase (FetchSearchPhase) by first reducing the results.
*/
Expand Down Expand Up @@ -88,7 +88,7 @@ public void onFailure(Exception e) {

void innerRun() throws Exception {
// if the RankBuilder specifies a QueryPhaseCoordinatorContext, it will be called as part of the reduce call
// to operate on the first `window_size * num_shards` results and merge them appropriately.
// to operate on the first `rank_window_size * num_shards` results and merge them appropriately.
SearchPhaseController.ReducedQueryPhase reducedQueryPhase = queryPhaseResults.reduce();
RankFeaturePhaseRankCoordinatorContext rankFeaturePhaseRankCoordinatorContext = coordinatorContext(context.getRequest().source());
if (rankFeaturePhaseRankCoordinatorContext != null) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
/**
* {@link QueryPhaseRankCoordinatorContext} is running on the coordinator node and is
* responsible for combining the query phase results from the shards and rank them accordingly.
* The output is a `window_size` ranked list of ordered results from all shards.
* The output is a `rank_window_size` ranked list of ordered results from all shards.
* Note: Currently this can use only sort by score; sort by field is not supported.
*/
public abstract class QueryPhaseRankCoordinatorContext {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
import java.util.List;

/**
* {@link QueryPhaseRankShardContext} is used to generate the top {@code window_size}
* {@link QueryPhaseRankShardContext} is used to generate the top {@code rank_window_size}
* results on each shard. It specifies the queries to run during {@code QueryPhase} and is responsible for combining all query scores and
* order all results through the {@link QueryPhaseRankShardContext#combineQueryPhaseResults} method.
*/
Expand Down
Loading