Fix to properly account min_score when having multiple nested retrievers (elastic#142212) (elastic#142365)

Author: pmpailis

docs/reference/elasticsearch/rest-apis/retrievers.md

@@ -57,6 +57,16 @@ The [`from`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operati
 [Aggregations](/reference/aggregations/index.md) are globally specified as part of a search request. The query used for an aggregation is the combination of all leaf retrievers as `should` clauses in a [boolean query](/reference/query-languages/query-dsl/query-dsl-bool-query.md).
 
 
+### Using `min_score` with compound retrievers [retriever-min-score-compound]
+
+When using `min_score` with compound retrievers (such as [`rrf`](retrievers/rrf-retriever.md) or [`linear`](retrievers/linear-retriever.md)), documents are filtered **after** the compound scoring has been applied. This is important to understand because:
+
+1. **Document collection**: Each child retriever collects documents up to its `rank_window_size` limit.
+2. **Score computation**: The compound retriever computes final scores (e.g., RRF ranking, linear combination with normalization).
+3. **Threshold filtering**: Documents with a final score below `min_score` are excluded from the results.
+
+Because `min_score` is applied after score normalization or RRF computation, the `total_hits` value reflects only the documents that pass the threshold after the compound scoring, not the total number of documents matched by the child retrievers.
+
 ### Restrictions on search parameters when specifying a retriever [retriever-restrictions]
 
 When a retriever is specified as part of a search, the following elements are not allowed at the top-level:

docs/reference/elasticsearch/rest-apis/retrievers/retrievers-examples.md

@@ -199,7 +199,7 @@ This returns the following response based on the final rrf score for each result
 ::::
 
 
-### Using the expanded format with weights 
+### Using the expanded format with weights
 ```{applies_to}
 stack: ga 9.2
 ```
@@ -1405,6 +1405,109 @@ GET retrievers_example/_search
 
 
 
+## Example: Multi-level nested retrievers with min_score [retrievers-examples-nested-retrievers-min-score]
+
+This example demonstrates how `min_score` works with deeply nested compound retrievers. The retriever tree structure is:
+
+- **Outer RRF** (rank_window_size: 20)
+  - **Inner RRF** (rank_window_size: 50)
+    - **Linear** (rank_window_size: 100, minmax normalizer, min_score: 0.5)
+      - **Standard** (query_string query)
+
+Documents are first matched by the innermost `standard` retriever using a query_string query. The `linear` retriever then normalizes these scores using the minmax normalizer and filters based on the specified `min_score`. Then, the inner `rrf` retriever computes the scores for the subset of documents that pass `min_score`, and finally the outer `rrf` retriever produces the final ranking.
+
+The `total_hits` value in this scenario is constrained by the `rank_window_size` parameters, as parent retrievers only have access to the top documents from their children.
+
+**Pagination behavior**: When using `from` and `size` at the top level of the search request, pagination is limited to the documents available at the outermost retriever's `rank_window_size`. In this example, even though the inner retrievers process more documents (100 for `linear`, 50 for inner `rrf`), the outer `rrf` only receives 50 documents and produces a final set of 20 documents. Therefore, `from` and `size` can only paginate through these top 20 documents.
+
+```console
+GET /retrievers_example/_search
+{
+    "retriever": {
+        "rrf": {
+            "retrievers": [
+                {
+                    "rrf": {
+                        "retrievers": [
+                            {
+                                "linear": {
+                                    "min_score": 0.5,
+                                    "retrievers": [
+                                        {
+                                            "retriever": {
+                                                "standard": {
+                                                    "query": {
+                                                        "query_string": {
+                                                            "query": "artificial intelligence",
+                                                            "default_field": "text"
+                                                        }
+                                                    }
+                                                }
+                                            }
+                                        }
+                                    ],
+                                    "rank_window_size": 100,
+                                    "normalizer": "minmax"
+                                }
+                            }
+                        ],
+                        "rank_window_size": 50
+                    }
+                }
+            ],
+            "rank_window_size": 20
+        }
+    },
+    "size": 10
+}
+```
+% TEST[continued]
+
+In this example:
+1. The `standard` retriever matches documents containing "artificial intelligence" in the text field.
+2. The `linear` retriever normalizes these scores to a 0-1 range using minmax normalization and filters documents with score less than 0.5.
+3. The inner `rrf` computes RRF scores based on document ranks.
+4. The outer `rrf` produces the final RRF scores.
+5. The `total_hits` value reflects only the documents passing the inner `min_score` threshold.
+
+::::{dropdown} Example response
+```console-result
+{
+    "took": 42,
+    "timed_out": false,
+    "_shards": {
+        "total": 1,
+        "successful": 1,
+        "skipped": 0,
+        "failed": 0
+    },
+    "hits": {
+        "total": {
+            "value": 1,
+            "relation": "eq"
+        },
+        "max_score": 0.016393442,
+        "hits": [
+            {
+                "_index": "retrievers_example",
+                "_id": "2",
+                "_score": 0.016393442,
+                "_source": {
+                    "vector": [0.12, 0.56, 0.78],
+                    "text": "Artificial intelligence is transforming medicine, from advancing diagnostics and tailoring treatment plans to empowering predictive patient care for improved health outcomes.",
+                    "year": 2023,
+                    "topic": ["ai", "medicine"],
+                    "timestamp": "2022-01-01T12:10:30"
+                }
+            }
+        ]
+    }
+}
+```
+% TESTRESPONSE[s/"took": 42/"took" : $body.took/]
+::::
+
+
 ## Example: Explainability with multiple retrievers [retrievers-examples-explain-multiple-rrf]
 
 By adding `explain: true` to the request, each retriever will now provide a detailed explanation of all the steps and calculations required to compute the final score. Composability is fully supported in the context of `explain`, and each retriever will provide its own explanation, as shown in the example below.

server/src/main/java/org/elasticsearch/search/retriever/RankDocsRetrieverBuilder.java

@@ -9,6 +9,7 @@
 
 package org.elasticsearch.search.retriever;
 
+import org.elasticsearch.features.NodeFeature;
 import org.elasticsearch.index.query.BoolQueryBuilder;
 import org.elasticsearch.index.query.QueryBuilder;
 import org.elasticsearch.index.query.QueryRewriteContext;
@@ -29,6 +30,10 @@
  */
 public class RankDocsRetrieverBuilder extends RetrieverBuilder {
 
+    public static final NodeFeature NESTED_RETRIEVER_MIN_SCORE_TOTAL_HITS_FIX = new NodeFeature(
+        "nested_retriever_min_score_total_hits_fix"
+    );
+
     public static final String NAME = "rank_docs_retriever";
     final int rankWindowSize;
     final List<RetrieverBuilder> sources;
@@ -49,8 +54,9 @@ public String getName() {
         return NAME;
     }
 
-    private boolean sourceHasMinScore() {
-        return this.minScore != null || sources.stream().anyMatch(x -> x.minScore() != null);
+    @Override
+    protected boolean hasMinScore() {
+        return this.minScore != null || sources.stream().anyMatch(RetrieverBuilder::hasMinScore);
     }
 
     private boolean sourceShouldRewrite(QueryRewriteContext ctx) throws IOException {
@@ -125,15 +131,15 @@ public void extractToSearchSourceBuilder(SearchSourceBuilder searchSourceBuilder
                 );
             }
         } else {
-            rankQuery = new RankDocsQueryBuilder(rankDocResults, null, false);
+            rankQuery = new RankDocsQueryBuilder(rankDocResults, null, true);
         }
         rankQuery.queryName(retrieverName());
         // ignore prefilters of this level, they were already propagated to children
         searchSourceBuilder.query(rankQuery);
         if (searchSourceBuilder.size() < 0) {
             searchSourceBuilder.size(rankWindowSize);
         }
-        if (sourceHasMinScore()) {
+        if (hasMinScore()) {
             searchSourceBuilder.minScore(this.minScore == null ? Float.MIN_VALUE : this.minScore);
         }
 

server/src/main/java/org/elasticsearch/search/retriever/RetrieverBuilder.java

@@ -313,5 +313,9 @@ public String retrieverName() {
         return retrieverName;
     }
 
+    protected boolean hasMinScore() {
+        return this.minScore != null;
+    }
+
     // ---- END FOR TESTING ----
 }

x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/RankRRFFeatures.java

@@ -15,6 +15,7 @@
 import java.util.Set;
 
 import static org.elasticsearch.search.retriever.CompoundRetrieverBuilder.INNER_RETRIEVERS_FILTER_SUPPORT;
+import static org.elasticsearch.search.retriever.RankDocsRetrieverBuilder.NESTED_RETRIEVER_MIN_SCORE_TOTAL_HITS_FIX;
 import static org.elasticsearch.xpack.rank.linear.L2ScoreNormalizer.LINEAR_RETRIEVER_L2_NORM;
 import static org.elasticsearch.xpack.rank.linear.LinearRetrieverBuilder.LINEAR_RETRIEVER_MINSCORE_FIX;
 import static org.elasticsearch.xpack.rank.linear.MinMaxScoreNormalizer.LINEAR_RETRIEVER_MINMAX_SINGLE_DOC_FIX;
@@ -42,7 +43,8 @@ public Set<NodeFeature> getTestFeatures() {
             RRFRetrieverBuilder.SIMPLIFIED_WEIGHTED_SUPPORT,
             LINEAR_RETRIEVER_TOP_LEVEL_NORMALIZER,
             LinearRetrieverBuilder.MULTI_INDEX_SIMPLIFIED_FORMAT_SUPPORT,
-            RRFRetrieverBuilder.MULTI_INDEX_SIMPLIFIED_FORMAT_SUPPORT
+            RRFRetrieverBuilder.MULTI_INDEX_SIMPLIFIED_FORMAT_SUPPORT,
+            NESTED_RETRIEVER_MIN_SCORE_TOTAL_HITS_FIX
         );
     }
 }

x-pack/plugin/rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/rrf/700_rrf_retriever_search_api_compatibility.yml

@@ -1148,3 +1148,48 @@ setup:
   - match: { hits.hits.1._id: "3" }
 
 
+---
+"nested compound retrievers with min_score correctly compute total_hits":
+  - requires:
+      cluster_features: "nested_retriever_min_score_total_hits_fix"
+      reason: "requires fix for correct total_hits computation with nested min_score"
+
+  - do:
+      search:
+        index: test
+        body:
+          retriever:
+            rrf:
+              retrievers:
+                - rrf:
+                    retrievers:
+                      - linear:
+                          retrievers: [
+                            {
+                              retriever: {
+                                standard: {
+                                  query: {
+                                    query_string: {
+                                      query: "term1 OR term2 OR term3",
+                                      default_field: "text"
+                                    }
+                                  }
+                                }
+                              }
+                            }
+                          ]
+                          rank_window_size: 100
+                          min_score: 0.5
+                          normalizer: "minmax"
+                    rank_window_size: 100
+              rank_window_size: 100
+          size: 10
+
+  - match: { hits.total.value: 3 }
+  - match: { hits.total.relation: "eq" }
+  - length: { hits.hits: 3 }
+  - match: { hits.hits.0._id: "1" }
+  - match: { hits.hits.1._id: "2" }
+  - match: { hits.hits.2._id: "3" }
+
+