Add docs for chunk_rescorer in text_similarity_reranker (#136428)

* Add docs for chunk_rescorer

* Updates to docs based on preview

* Update docs/reference/elasticsearch/rest-apis/retrievers/text-similarity-reranker-retriever.md

Co-authored-by: Liam Thompson <[email protected]>

* Update docs/reference/elasticsearch/rest-apis/retrievers/text-similarity-reranker-retriever.md

Co-authored-by: Liam Thompson <[email protected]>

* Update docs/reference/elasticsearch/rest-apis/retrievers/text-similarity-reranker-retriever.md

Co-authored-by: Liam Thompson <[email protected]>

---------

Co-authored-by: Liam Thompson <[email protected]>

Author: kderusso

docs/reference/elasticsearch/rest-apis/retrievers/retrievers-examples.md

@@ -440,7 +440,7 @@ GET /retrievers_example/_search
             "query": "artificial intelligence"
         }
     }
-}     
+}
 ```
 
 This returns the following response based on the final rrf score for each result.
@@ -497,7 +497,7 @@ GET /retrievers_example/_search
             "fields": ["text", "text_semantic"]
         }
     }
-}     
+}
 ```
 
 ::::{note}
@@ -570,7 +570,7 @@ GET /retrievers_example/_search
             "normalizer": "minmax"
         }
     }
-}     
+}
 ```
 
 This returns the following response based on the normalized score for each result:
@@ -1503,6 +1503,7 @@ PUT _inference/rerank/my-rerank-model
 ```
 
 Let’s start by reranking the results of the `rrf` retriever in our previous example.
+We'll also apply a `chunk_rescorer` to ensure that we only consider the best scoring chunks when sending information to the reranker.
 
 ```console
 GET retrievers_example/_search
@@ -1541,7 +1542,15 @@ GET retrievers_example/_search
             },
             "field": "text",
             "inference_id": "my-rerank-model",
-            "inference_text": "What are the state of the art applications of AI in information retrieval?"
+            "inference_text": "What are the state of the art applications of AI in information retrieval?",
+            "chunk_rescorer": {
+                "size": 1,
+                "chunking_settings": {
+                    "strategy": "sentence",
+                    "max_chunk_size": 300,
+                    "sentence_overlap": 0
+                }
+            },
         }
     },
     "_source": false

docs/reference/elasticsearch/rest-apis/retrievers/text-similarity-reranker-retriever.md

@@ -86,6 +86,25 @@ score = ln(score), if score < 0
 
     Applies the specified [boolean query filter](/reference/query-languages/query-dsl/query-dsl-bool-query.md) to the child  `retriever`. If the child retriever already specifies any filters, then this top-level filter is applied in conjuction with the filter defined in the child retriever.
 
+`chunk_rescorer` {applies_to}`stack: beta 9.2`
+:   (Optional, `object`)
+
+    Chunks and scores documents based on configured chunking settings, and only sends the best scoring chunks to the reranking model as input. This helps improve relevance when reranking long documents that would otherwise be truncated by the reranking model's token limit.
+
+    Parameters for `chunk_rescorer`:
+
+    `size`
+    :   (Optional, `int`)
+
+    The number of chunks to pass to the reranker. Defaults to `1`.
+
+    `chunking_settings`
+    :   (Optional, `object`)
+
+    Settings for chunking text into smaller passages for scoring and reranking. Defaults to the optimal chunking settings for [Elastic Rerank](docs-content:///explore-analyze/machine-learning/nlp/ml-nlp-rerank.md). Refer to the [Inference API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put#operation-inference-put-body-application-json-chunking_settings) for valid values for `chunking_settings`. 
+    :::{warning} 
+    If you configure chunks larger than the reranker's token limit, the results may be truncated. This can degrade relevance significantly.
+    :::
 
 
 ## Example: Elastic Rerank [text-similarity-reranker-retriever-example-elastic-rerank]