From 1c91bb3be9e5e9206f64bf2fa18aee381d5ca537 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Mon, 16 Sep 2024 18:41:00 +0200 Subject: [PATCH 01/12] [DOCS] Fix reranking IA, move retrievers to search api overview --- .../search-your-data/learning-to-rank.asciidoc | 14 +++++++------- .../search-your-data/reranking/index.asciidoc | 16 ++++++++++++++++ .../semantic-reranking.asciidoc | 4 ---- .../retrievers-overview.asciidoc | 0 .../retrievers-reranking/index.asciidoc | 8 -------- .../search/search-your-data/search-api.asciidoc | 2 ++ .../search-your-data/search-your-data.asciidoc | 3 +-- 7 files changed, 26 insertions(+), 21 deletions(-) create mode 100644 docs/reference/search/search-your-data/reranking/index.asciidoc rename docs/reference/search/search-your-data/{retrievers-reranking => reranking}/semantic-reranking.asciidoc (95%) rename docs/reference/search/search-your-data/{retrievers-reranking => }/retrievers-overview.asciidoc (100%) delete mode 100644 docs/reference/search/search-your-data/retrievers-reranking/index.asciidoc diff --git a/docs/reference/search/search-your-data/learning-to-rank.asciidoc b/docs/reference/search/search-your-data/learning-to-rank.asciidoc index ebd6d67fe42da..e08d5e5dc55dc 100644 --- a/docs/reference/search/search-your-data/learning-to-rank.asciidoc +++ b/docs/reference/search/search-your-data/learning-to-rank.asciidoc @@ -1,5 +1,5 @@ [[learning-to-rank]] -== Learning To Rank +=== Learning To Rank (LTR) NOTE: This feature was introduced in version 8.12.0 and is only available to certain subscription levels. For more information, see {subscriptions}. @@ -17,7 +17,7 @@ image::images/search/learning-to-rank-overview.png[Learning To Rank overview,ali [discrete] [[learning-to-rank-search-context]] -=== Search context +==== Search context In addition to the list of documents to sort, the LTR function also requires a search context. Typically, this search context includes at least the search @@ -27,7 +27,7 @@ This could be information about the user doing the search (such as demographic d [discrete] [[learning-to-rank-judgement-list]] -=== Judgment list +==== Judgment list The LTR model is usually trained on a judgment list, which is a set of queries and documents with a relevance grade. Judgment lists can be human or machine generated: they're commonly populated from behavioural analytics, often with human moderation. Judgment lists determine the ideal ordering of results for a given search query. The goal of LTR is to fit the model to the judgment list rankings as closely as possible for new queries and documents. The judgment list is the main input used to train the model. It consists of a dataset that contains pairs of queries and documents, along with their corresponding relevance labels. @@ -42,7 +42,7 @@ image::images/search/learning-to-rank-judgment-list.png[Judgment list example,al [discrete] [[judgment-list-notes]] -==== Notes on judgment lists +===== Notes on judgment lists While a judgment list can be created manually by humans, there are techniques available to leverage user engagement data, such as clicks or conversions, to construct judgment lists automatically. @@ -54,7 +54,7 @@ The quantity and the quality of your judgment list will greatly influence the ov [discrete] [[learning-to-rank-feature-extraction]] -=== Feature extraction +==== Feature extraction Query and document pairs alone don't provide enough information to train the ML models used for LTR. The relevance scores in judgment lists depend on a number @@ -101,7 +101,7 @@ templated query: [discrete] [[learning-to-rank-models]] -=== Models +==== Models The heart of LTR is of course an ML model. A model is trained using the training data described above in combination with an objective. In the case of LTR, the objective is to rank result documents in an optimal way with respect to a judgment list, given some ranking metric such as https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Discounted_cumulative_gain[nDCG^] or https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Mean_average_precision[MAP^]. The model relies solely on the features and relevance labels from the training data. @@ -123,7 +123,7 @@ Learn more about training in <>: Use machine learning models to reorder search results based on their semantic similarity to a query. Models can be hosted directly in your {es} cluster, or you can use <> to use models provided by third-party services. Enables out-of-the-box semantic search capabilities on existing indices. +* <>: Train a machine learning model to build a ranking function for your search experience that updates over time. For advanced use cases. + +include::semantic-reranking.asciidoc[] +include::../learning-to-rank.asciidoc[] diff --git a/docs/reference/search/search-your-data/retrievers-reranking/semantic-reranking.asciidoc b/docs/reference/search/search-your-data/reranking/semantic-reranking.asciidoc similarity index 95% rename from docs/reference/search/search-your-data/retrievers-reranking/semantic-reranking.asciidoc rename to docs/reference/search/search-your-data/reranking/semantic-reranking.asciidoc index add2d7455983e..a11cc445d581e 100644 --- a/docs/reference/search/search-your-data/retrievers-reranking/semantic-reranking.asciidoc +++ b/docs/reference/search/search-your-data/reranking/semantic-reranking.asciidoc @@ -11,10 +11,6 @@ This overview focuses more on the high-level concepts and use cases for semantic Rerankers improve the relevance of results from earlier-stage retrieval mechanisms. _Semantic_ rerankers use machine learning models to reorder search results based on their semantic similarity to a query. -First-stage retrievers and rankers must be very fast and efficient because they process either the entire corpus, or all matching documents. -In a multi-stage pipeline, you can progressively use more computationally intensive ranking functions and techniques, as they will operate on smaller result sets at each step. -This helps avoid query latency degradation and keeps costs manageable. - Semantic reranking requires relatively large and complex machine learning models and operates in real-time in response to queries. This technique makes sense on a small _top-k_ result set, as one the of the final steps in a pipeline. This is a powerful technique for improving search relevance that works equally well with keyword, semantic, or hybrid retrieval algorithms. diff --git a/docs/reference/search/search-your-data/retrievers-reranking/retrievers-overview.asciidoc b/docs/reference/search/search-your-data/retrievers-overview.asciidoc similarity index 100% rename from docs/reference/search/search-your-data/retrievers-reranking/retrievers-overview.asciidoc rename to docs/reference/search/search-your-data/retrievers-overview.asciidoc diff --git a/docs/reference/search/search-your-data/retrievers-reranking/index.asciidoc b/docs/reference/search/search-your-data/retrievers-reranking/index.asciidoc deleted file mode 100644 index 87ed52e365370..0000000000000 --- a/docs/reference/search/search-your-data/retrievers-reranking/index.asciidoc +++ /dev/null @@ -1,8 +0,0 @@ -[[retrievers-reranking-overview]] -== Retrievers and reranking - -* <> -* <> - -include::retrievers-overview.asciidoc[] -include::semantic-reranking.asciidoc[] diff --git a/docs/reference/search/search-your-data/search-api.asciidoc b/docs/reference/search/search-your-data/search-api.asciidoc index 98c5a48b7559b..13cea537ea4fb 100644 --- a/docs/reference/search/search-your-data/search-api.asciidoc +++ b/docs/reference/search/search-your-data/search-api.asciidoc @@ -530,3 +530,5 @@ include::retrieve-inner-hits.asciidoc[] include::search-shard-routing.asciidoc[] include::search-using-query-rules.asciidoc[] include::search-template.asciidoc[] +include::retrievers-overview.asciidoc[] + diff --git a/docs/reference/search/search-your-data/search-your-data.asciidoc b/docs/reference/search/search-your-data/search-your-data.asciidoc index a885df2f2179e..70eccb4b022bc 100644 --- a/docs/reference/search/search-your-data/search-your-data.asciidoc +++ b/docs/reference/search/search-your-data/search-your-data.asciidoc @@ -45,8 +45,7 @@ results directly in the Kibana Search UI. include::search-api.asciidoc[] include::knn-search.asciidoc[] include::semantic-search.asciidoc[] -include::retrievers-reranking/index.asciidoc[] -include::learning-to-rank.asciidoc[] +include::reranking/index.asciidoc[] include::search-across-clusters.asciidoc[] include::search-with-synonyms.asciidoc[] include::search-application-overview.asciidoc[] From 1abad3835415a158a3a5bae815b0bcd6489de876 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Wed, 18 Sep 2024 18:05:37 +0200 Subject: [PATCH 02/12] Move to toplevel reranking section --- docs/reference/index.asciidoc | 2 ++ .../reranking/index.asciidoc | 4 ++-- .../learning-to-rank-model-training.asciidoc | 0 .../learning-to-rank-search-usage.asciidoc | 0 .../learning-to-rank.asciidoc | 16 ++++++++-------- .../reranking/semantic-reranking.asciidoc | 10 +++++----- .../search-your-data/search-your-data.asciidoc | 1 - 7 files changed, 17 insertions(+), 16 deletions(-) rename docs/reference/{search/search-your-data => }/reranking/index.asciidoc (95%) rename docs/reference/{search/search-your-data => reranking}/learning-to-rank-model-training.asciidoc (100%) rename docs/reference/{search/search-your-data => reranking}/learning-to-rank-search-usage.asciidoc (100%) rename docs/reference/{search/search-your-data => reranking}/learning-to-rank.asciidoc (97%) rename docs/reference/{search/search-your-data => }/reranking/semantic-reranking.asciidoc (98%) diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index 2057519719177..79b5f2b69f24d 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -32,6 +32,8 @@ include::alias.asciidoc[] include::search/search-your-data/search-your-data.asciidoc[] +include::reranking/index.asciidoc[] + include::query-dsl.asciidoc[] include::aggregations.asciidoc[] diff --git a/docs/reference/search/search-your-data/reranking/index.asciidoc b/docs/reference/reranking/index.asciidoc similarity index 95% rename from docs/reference/search/search-your-data/reranking/index.asciidoc rename to docs/reference/reranking/index.asciidoc index 318485d9e88f7..92778beb1b96f 100644 --- a/docs/reference/search/search-your-data/reranking/index.asciidoc +++ b/docs/reference/reranking/index.asciidoc @@ -1,5 +1,5 @@ [[reranking-overview]] -== Reranking += Reranking Rerankers improve the relevance of results from earlier-stage retrieval mechanisms. First-stage rankers must be very fast and efficient because they process either the entire corpus, or all matching documents. @@ -13,4 +13,4 @@ Currently, the following reranking techniques are available in {es}: * <>: Train a machine learning model to build a ranking function for your search experience that updates over time. For advanced use cases. include::semantic-reranking.asciidoc[] -include::../learning-to-rank.asciidoc[] +include::learning-to-rank.asciidoc[] diff --git a/docs/reference/search/search-your-data/learning-to-rank-model-training.asciidoc b/docs/reference/reranking/learning-to-rank-model-training.asciidoc similarity index 100% rename from docs/reference/search/search-your-data/learning-to-rank-model-training.asciidoc rename to docs/reference/reranking/learning-to-rank-model-training.asciidoc diff --git a/docs/reference/search/search-your-data/learning-to-rank-search-usage.asciidoc b/docs/reference/reranking/learning-to-rank-search-usage.asciidoc similarity index 100% rename from docs/reference/search/search-your-data/learning-to-rank-search-usage.asciidoc rename to docs/reference/reranking/learning-to-rank-search-usage.asciidoc diff --git a/docs/reference/search/search-your-data/learning-to-rank.asciidoc b/docs/reference/reranking/learning-to-rank.asciidoc similarity index 97% rename from docs/reference/search/search-your-data/learning-to-rank.asciidoc rename to docs/reference/reranking/learning-to-rank.asciidoc index e08d5e5dc55dc..f49f7074d2dbc 100644 --- a/docs/reference/search/search-your-data/learning-to-rank.asciidoc +++ b/docs/reference/reranking/learning-to-rank.asciidoc @@ -1,5 +1,5 @@ [[learning-to-rank]] -=== Learning To Rank (LTR) +== Learning To Rank NOTE: This feature was introduced in version 8.12.0 and is only available to certain subscription levels. For more information, see {subscriptions}. @@ -17,7 +17,7 @@ image::images/search/learning-to-rank-overview.png[Learning To Rank overview,ali [discrete] [[learning-to-rank-search-context]] -==== Search context +=== Search context In addition to the list of documents to sort, the LTR function also requires a search context. Typically, this search context includes at least the search @@ -27,7 +27,7 @@ This could be information about the user doing the search (such as demographic d [discrete] [[learning-to-rank-judgement-list]] -==== Judgment list +=== Judgment list The LTR model is usually trained on a judgment list, which is a set of queries and documents with a relevance grade. Judgment lists can be human or machine generated: they're commonly populated from behavioural analytics, often with human moderation. Judgment lists determine the ideal ordering of results for a given search query. The goal of LTR is to fit the model to the judgment list rankings as closely as possible for new queries and documents. The judgment list is the main input used to train the model. It consists of a dataset that contains pairs of queries and documents, along with their corresponding relevance labels. @@ -42,7 +42,7 @@ image::images/search/learning-to-rank-judgment-list.png[Judgment list example,al [discrete] [[judgment-list-notes]] -===== Notes on judgment lists +==== Notes on judgment lists While a judgment list can be created manually by humans, there are techniques available to leverage user engagement data, such as clicks or conversions, to construct judgment lists automatically. @@ -54,7 +54,7 @@ The quantity and the quality of your judgment list will greatly influence the ov [discrete] [[learning-to-rank-feature-extraction]] -==== Feature extraction +=== Feature extraction Query and document pairs alone don't provide enough information to train the ML models used for LTR. The relevance scores in judgment lists depend on a number @@ -101,7 +101,7 @@ templated query: [discrete] [[learning-to-rank-models]] -==== Models +=== Models The heart of LTR is of course an ML model. A model is trained using the training data described above in combination with an objective. In the case of LTR, the objective is to rank result documents in an optimal way with respect to a judgment list, given some ranking metric such as https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Discounted_cumulative_gain[nDCG^] or https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Mean_average_precision[MAP^]. The model relies solely on the features and relevance labels from the training data. @@ -123,7 +123,7 @@ Learn more about training in <> include::learning-to-rank-model-training.asciidoc[] -include::learning-to-rank-search-usage.asciidoc[] +include::learning-to-rank-search-usage.asciidoc[] \ No newline at end of file diff --git a/docs/reference/search/search-your-data/reranking/semantic-reranking.asciidoc b/docs/reference/reranking/semantic-reranking.asciidoc similarity index 98% rename from docs/reference/search/search-your-data/reranking/semantic-reranking.asciidoc rename to docs/reference/reranking/semantic-reranking.asciidoc index a11cc445d581e..200a5f673dc0c 100644 --- a/docs/reference/search/search-your-data/reranking/semantic-reranking.asciidoc +++ b/docs/reference/reranking/semantic-reranking.asciidoc @@ -1,5 +1,5 @@ [[semantic-reranking]] -=== Semantic reranking +== Semantic reranking preview::[] @@ -20,7 +20,7 @@ The final sections include a practical, high-level overview of how to implement [discrete] [[semantic-reranking-use-cases]] -==== Use cases +=== Use cases Semantic reranking enables a variety of use cases: @@ -41,7 +41,7 @@ Now that we've outlined the value of semantic reranking, we'll explore the speci [discrete] [[semantic-reranking-models]] -==== Cross-encoder and bi-encoder models +=== Cross-encoder and bi-encoder models At a high level, two model types are used for semantic reranking: cross-encoders and bi-encoders. @@ -79,7 +79,7 @@ Note that similarity scores from bi-encoders/embedding similarities are _query-d [discrete] [[semantic-reranking-in-es]] -==== Semantic reranking in {es} +=== Semantic reranking in {es} In {es}, semantic rerankers are implemented using the {es} <> and a <>. @@ -130,7 +130,7 @@ POST _search [discrete] [[semantic-reranking-learn-more]] -==== Learn more +=== Learn more * Read the <> for syntax and implementation details * Learn more about the <> abstraction diff --git a/docs/reference/search/search-your-data/search-your-data.asciidoc b/docs/reference/search/search-your-data/search-your-data.asciidoc index 70eccb4b022bc..cd2b418a7e79b 100644 --- a/docs/reference/search/search-your-data/search-your-data.asciidoc +++ b/docs/reference/search/search-your-data/search-your-data.asciidoc @@ -45,7 +45,6 @@ results directly in the Kibana Search UI. include::search-api.asciidoc[] include::knn-search.asciidoc[] include::semantic-search.asciidoc[] -include::reranking/index.asciidoc[] include::search-across-clusters.asciidoc[] include::search-with-synonyms.asciidoc[] include::search-application-overview.asciidoc[] From 1bc2a297feb777b4c56cede6a9217e50c3fcbc70 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Thu, 19 Sep 2024 11:59:34 +0200 Subject: [PATCH 03/12] Provide a better overview of ranking strategies --- docs/reference/reranking/index.asciidoc | 57 ++++++++++++++++++++++--- 1 file changed, 51 insertions(+), 6 deletions(-) diff --git a/docs/reference/reranking/index.asciidoc b/docs/reference/reranking/index.asciidoc index 92778beb1b96f..52f088fd3f470 100644 --- a/docs/reference/reranking/index.asciidoc +++ b/docs/reference/reranking/index.asciidoc @@ -2,15 +2,60 @@ = Reranking Rerankers improve the relevance of results from earlier-stage retrieval mechanisms. -First-stage rankers must be very fast and efficient because they process either the entire corpus, or all matching documents. +{es} supports a number of <>. -In a multi-stage pipeline, you can progressively use more computationally intensive ranking functions and techniques, as they will operate on smaller result sets at each step. -This helps avoid query latency degradation and keeps costs manageable. +We provide a brief overview of ranking mechanisms below, from the simplest to the most complex, to help you choose the right workflow for your use case. -Currently, the following reranking techniques are available in {es}: +[float] +[[reranking-ranking-overview-bm25]] +== Full-text search: BM25 scoring -* <>: Use machine learning models to reorder search results based on their semantic similarity to a query. Models can be hosted directly in your {es} cluster, or you can use <> to use models provided by third-party services. Enables out-of-the-box semantic search capabilities on existing indices. -* <>: Train a machine learning model to build a ranking function for your search experience that updates over time. For advanced use cases. +{es} ranks documents based on term frequency and inverse document frequency, adjusted for document length. + +BM25 is the default statistical scoring algorithm in {es} and works out-of-the-box. + +[float] +[[reranking-ranking-overview-vector]] +== Vector search: similarity scoring + +Vector search involves transforming data into dense or sparse vector embeddings to capture semantic meanings, and computing similarity scores for query vectors. + +Store vectors using `semantic` fields for automatic vectorization or `dense_vector` and `sparse_vector` fields when you need more control over the underlying embedding model. + +Query vectors with `semantic` queries or `knn` and `sparse_vector` queries to compute similarity scores. + +Refer to <> for more information. + +[float] +[[reranking-ranking-overview-hybrid]] +== Hybrid techniques + +Hybrid search techniques combine results from full-text and vector search pipelines. + +{es} enables combining lexical matching (BM25) and vector search scores using the advanced <> algorithm. + +[float] +[[reranking-ranking-overview-advanced]] +== Advanced reranking + +When using the following advanced reranking pipelines, early retrieval mechanisms effectively generate a set of candidates. +These candidates are funneled into a late stage reranker that performs more computationally expensive reranking tasks. + +[float] +[[reranking-ranking-overview-semantic]] +=== Semantic reranking + +<> uses machine learning models to reorder search results based on their semantic similarity to a query. + +Models can be hosted directly in your {es} cluster, or you can use <> to use models provided by third-party services. Enables out-of-the-box semantic search capabilities on existing full-text search indices. + +[float] +[[reranking-ranking-overview-ltr]] +=== Learning to Rank (LTR) + +<> is for advanced users. Train a machine learning model to build a ranking function for your search experience that updates over time. + +Best suited for when you have ample training data and need highly customized relevance tuning. include::semantic-reranking.asciidoc[] include::learning-to-rank.asciidoc[] From 72359f29c2d50720fbdcc8d133eaca4e16755f9b Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Thu, 19 Sep 2024 12:03:59 +0200 Subject: [PATCH 04/12] Nit rewording --- docs/reference/reranking/index.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/reranking/index.asciidoc b/docs/reference/reranking/index.asciidoc index 52f088fd3f470..0b4f4bdf855f7 100644 --- a/docs/reference/reranking/index.asciidoc +++ b/docs/reference/reranking/index.asciidoc @@ -47,7 +47,7 @@ These candidates are funneled into a late stage reranker that performs more comp <> uses machine learning models to reorder search results based on their semantic similarity to a query. -Models can be hosted directly in your {es} cluster, or you can use <> to use models provided by third-party services. Enables out-of-the-box semantic search capabilities on existing full-text search indices. +Models can be hosted directly in your {es} cluster, or you can use <> to call models provided by third-party services. Enables out-of-the-box semantic search capabilities on existing full-text search indices. [float] [[reranking-ranking-overview-ltr]] From ba1027d1c785fb5fa42eeede47aa298ab2c180b5 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Thu, 19 Sep 2024 13:45:57 +0200 Subject: [PATCH 05/12] Reword to focus on two-stage pipelines --- docs/reference/reranking/index.asciidoc | 68 ++++++++++++++----------- 1 file changed, 39 insertions(+), 29 deletions(-) diff --git a/docs/reference/reranking/index.asciidoc b/docs/reference/reranking/index.asciidoc index 0b4f4bdf855f7..fe6d57ebde91c 100644 --- a/docs/reference/reranking/index.asciidoc +++ b/docs/reference/reranking/index.asciidoc @@ -1,61 +1,71 @@ -[[reranking-overview]] -= Reranking +[[re-ranking-overview]] += Re-ranking -Rerankers improve the relevance of results from earlier-stage retrieval mechanisms. -{es} supports a number of <>. +Many search systems are built on two-stage retrieval pipelines. -We provide a brief overview of ranking mechanisms below, from the simplest to the most complex, to help you choose the right workflow for your use case. +The first stage uses cheap, fast algorithms to find a subset of possible matches. + +The second stage uses a more powerful model, often machine learning-based, to reorder the documents. +This second step is called re-ranking. +This approach balances computational costs, because the resource-intensive model is only applied to a smaller set of pre-filtered results. + +{es} supports various ranking and re-ranking techniques to optimize search relevance and performance. [float] -[[reranking-ranking-overview-bm25]] -== Full-text search: BM25 scoring +[[re-ranking-two-stage-pipeline]] +== Two-stage retrieval pipelines -{es} ranks documents based on term frequency and inverse document frequency, adjusted for document length. +Learn about retrieval pipelines and how re-ranking can be applied to your existing search experience. + +[float] +[[re-ranking-first-stage-pipeline]] +=== First stage: initial retrieval + +[float] +[[re-ranking-ranking-overview-bm25]] +==== Full-text search: BM25 scoring +{es} ranks documents based on term frequency and inverse document frequency, adjusted for document length. BM25 is the default statistical scoring algorithm in {es} and works out-of-the-box. [float] -[[reranking-ranking-overview-vector]] -== Vector search: similarity scoring +[[re-ranking-ranking-overview-vector]] +==== Vector search: similarity scoring Vector search involves transforming data into dense or sparse vector embeddings to capture semantic meanings, and computing similarity scores for query vectors. - Store vectors using `semantic` fields for automatic vectorization or `dense_vector` and `sparse_vector` fields when you need more control over the underlying embedding model. - Query vectors with `semantic` queries or `knn` and `sparse_vector` queries to compute similarity scores. - Refer to <> for more information. [float] -[[reranking-ranking-overview-hybrid]] -== Hybrid techniques +[[re-ranking-ranking-overview-hybrid]] +==== Hybrid techniques Hybrid search techniques combine results from full-text and vector search pipelines. - {es} enables combining lexical matching (BM25) and vector search scores using the advanced <> algorithm. [float] -[[reranking-ranking-overview-advanced]] -== Advanced reranking +[[re-ranking-overview-second-stage]] +=== Second Stage: Re-ranking -When using the following advanced reranking pipelines, early retrieval mechanisms effectively generate a set of candidates. -These candidates are funneled into a late stage reranker that performs more computationally expensive reranking tasks. +When using the following advanced re-ranking pipelines, early retrieval mechanisms effectively generate a set of candidates. +These candidates are funneled into a late stage reranker that performs more computationally expensive re-ranking tasks. [float] -[[reranking-ranking-overview-semantic]] -=== Semantic reranking +[[re-ranking-overview-semantic]] +==== Semantic re-ranking <> uses machine learning models to reorder search results based on their semantic similarity to a query. - -Models can be hosted directly in your {es} cluster, or you can use <> to call models provided by third-party services. Enables out-of-the-box semantic search capabilities on existing full-text search indices. +Models can be hosted directly in your {es} cluster, or you can use <> to call models provided by third-party services. +Enables out-of-the-box semantic search capabilities on existing full-text search indices. [float] -[[reranking-ranking-overview-ltr]] -=== Learning to Rank (LTR) - -<> is for advanced users. Train a machine learning model to build a ranking function for your search experience that updates over time. +[[re-ranking-overview-ltr]] +==== Learning to Rank (LTR) +<> is for advanced users. +Train a machine learning model to build a ranking function for your search experience that updates over time. Best suited for when you have ample training data and need highly customized relevance tuning. include::semantic-reranking.asciidoc[] -include::learning-to-rank.asciidoc[] +include::learning-to-rank.asciidoc[] \ No newline at end of file From 42e604267182703a69210ea51f83de4299064d71 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Thu, 19 Sep 2024 13:48:08 +0200 Subject: [PATCH 06/12] Nit copyedit --- docs/reference/reranking/index.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/reranking/index.asciidoc b/docs/reference/reranking/index.asciidoc index fe6d57ebde91c..6138e079ba442 100644 --- a/docs/reference/reranking/index.asciidoc +++ b/docs/reference/reranking/index.asciidoc @@ -48,8 +48,8 @@ Hybrid search techniques combine results from full-text and vector search pipeli [[re-ranking-overview-second-stage]] === Second Stage: Re-ranking -When using the following advanced re-ranking pipelines, early retrieval mechanisms effectively generate a set of candidates. -These candidates are funneled into a late stage reranker that performs more computationally expensive re-ranking tasks. +When using the following advanced re-ranking pipelines, first-stage retrieval mechanisms effectively generate a set of candidates. +These candidates are funneled into the re-ranker to perform more computationally expensive re-ranking tasks. [float] [[re-ranking-overview-semantic]] From ff7106b72d67d8ad60851aa0f8cc044e94b258ab Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Thu, 19 Sep 2024 15:34:55 +0200 Subject: [PATCH 07/12] Apply suggestions from code review Co-authored-by: Kathleen DeRusso --- docs/reference/reranking/index.asciidoc | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/reference/reranking/index.asciidoc b/docs/reference/reranking/index.asciidoc index 6138e079ba442..d501b2c038f62 100644 --- a/docs/reference/reranking/index.asciidoc +++ b/docs/reference/reranking/index.asciidoc @@ -3,11 +3,11 @@ Many search systems are built on two-stage retrieval pipelines. -The first stage uses cheap, fast algorithms to find a subset of possible matches. +The first stage uses cheap, fast algorithms to find a broad set of possible matches. The second stage uses a more powerful model, often machine learning-based, to reorder the documents. This second step is called re-ranking. -This approach balances computational costs, because the resource-intensive model is only applied to a smaller set of pre-filtered results. +Because the resource-intensive model is only applied to the smaller set of pre-filtered results, this approach returns more relevant results while still optimizing for search performance and computational costs. {es} supports various ranking and re-ranking techniques to optimize search relevance and performance. @@ -15,7 +15,6 @@ This approach balances computational costs, because the resource-intensive model [[re-ranking-two-stage-pipeline]] == Two-stage retrieval pipelines -Learn about retrieval pipelines and how re-ranking can be applied to your existing search experience. [float] [[re-ranking-first-stage-pipeline]] @@ -26,7 +25,7 @@ Learn about retrieval pipelines and how re-ranking can be applied to your existi ==== Full-text search: BM25 scoring {es} ranks documents based on term frequency and inverse document frequency, adjusted for document length. -BM25 is the default statistical scoring algorithm in {es} and works out-of-the-box. +https://en.wikipedia.org/wiki/Okapi_BM25[BM25] is the default statistical scoring algorithm in {es} and works out-of-the-box. [float] [[re-ranking-ranking-overview-vector]] @@ -42,7 +41,7 @@ Refer to <> for more information. ==== Hybrid techniques Hybrid search techniques combine results from full-text and vector search pipelines. -{es} enables combining lexical matching (BM25) and vector search scores using the advanced <> algorithm. +{es} enables combining lexical matching (BM25) and vector search scores using the <> algorithm. [float] [[re-ranking-overview-second-stage]] @@ -57,15 +56,15 @@ These candidates are funneled into the re-ranker to perform more computationally <> uses machine learning models to reorder search results based on their semantic similarity to a query. Models can be hosted directly in your {es} cluster, or you can use <> to call models provided by third-party services. -Enables out-of-the-box semantic search capabilities on existing full-text search indices. +Semantic re-ranking enables out-of-the-box semantic search capabilities on existing full-text search indices. [float] [[re-ranking-overview-ltr]] ==== Learning to Rank (LTR) <> is for advanced users. -Train a machine learning model to build a ranking function for your search experience that updates over time. -Best suited for when you have ample training data and need highly customized relevance tuning. +Learning To Rank involves training a machine learning model to build a ranking function for your search experience that updates over time. +LTR is best suited for when you have ample training data and need highly customized relevance tuning. include::semantic-reranking.asciidoc[] include::learning-to-rank.asciidoc[] \ No newline at end of file From e815f9445911b6d5baebd65d156432b56287d80f Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Thu, 19 Sep 2024 15:46:12 +0200 Subject: [PATCH 08/12] Uniformize rerank -> re-rank --- .../reranking/semantic-reranking.asciidoc | 44 +++++++++---------- docs/reference/search/retriever.asciidoc | 8 ++-- 2 files changed, 26 insertions(+), 26 deletions(-) diff --git a/docs/reference/reranking/semantic-reranking.asciidoc b/docs/reference/reranking/semantic-reranking.asciidoc index 200a5f673dc0c..4ebe90e44708e 100644 --- a/docs/reference/reranking/semantic-reranking.asciidoc +++ b/docs/reference/reranking/semantic-reranking.asciidoc @@ -1,35 +1,35 @@ [[semantic-reranking]] -== Semantic reranking +== Semantic re-ranking preview::[] [TIP] ==== -This overview focuses more on the high-level concepts and use cases for semantic reranking. For full implementation details on how to set up and use semantic reranking in {es}, see the <> in the Search API docs. +This overview focuses more on the high-level concepts and use cases for semantic re-ranking. For full implementation details on how to set up and use semantic re-ranking in {es}, see the <> in the Search API docs. ==== -Rerankers improve the relevance of results from earlier-stage retrieval mechanisms. -_Semantic_ rerankers use machine learning models to reorder search results based on their semantic similarity to a query. +Re-rankers improve the relevance of results from earlier-stage retrieval mechanisms. +_Semantic_ re-rankers use machine learning models to reorder search results based on their semantic similarity to a query. -Semantic reranking requires relatively large and complex machine learning models and operates in real-time in response to queries. +Semantic re-ranking requires relatively large and complex machine learning models and operates in real-time in response to queries. This technique makes sense on a small _top-k_ result set, as one the of the final steps in a pipeline. This is a powerful technique for improving search relevance that works equally well with keyword, semantic, or hybrid retrieval algorithms. -The next sections provide more details on the benefits, use cases, and model types used for semantic reranking. -The final sections include a practical, high-level overview of how to implement <> and links to the full reference documentation. +The next sections provide more details on the benefits, use cases, and model types used for semantic re-ranking. +The final sections include a practical, high-level overview of how to implement <> and links to the full reference documentation. [discrete] [[semantic-reranking-use-cases]] === Use cases -Semantic reranking enables a variety of use cases: +Semantic re-ranking enables a variety of use cases: -* *Lexical (BM25) retrieval results reranking* +* *Lexical (BM25) retrieval results re-ranking* ** Out-of-the-box semantic search by adding a simple API call to any lexical/BM25 retrieval pipeline. ** Adds semantic search capabilities on top of existing indices without reindexing, perfect for quick improvements. ** Ideal for environments with complex existing indices. -* *Semantic retrieval results reranking* +* *Semantic retrieval results re-ranking* ** Improves results from semantic retrievers using ELSER sparse vector embeddings or dense vector embeddings by using more powerful models. ** Adds a refinement layer on top of hybrid retrieval with <>. @@ -37,15 +37,15 @@ Semantic reranking enables a variety of use cases: ** Supports automatic and transparent chunking, eliminating the need for pre-chunking at index time. ** Provides explicit control over document relevance in retrieval-augmented generation (RAG) uses cases or other scenarios involving language model (LLM) inputs. -Now that we've outlined the value of semantic reranking, we'll explore the specific models that power this process and how they differ. +Now that we've outlined the value of semantic re-ranking, we'll explore the specific models that power this process and how they differ. [discrete] [[semantic-reranking-models]] === Cross-encoder and bi-encoder models -At a high level, two model types are used for semantic reranking: cross-encoders and bi-encoders. +At a high level, two model types are used for semantic re-ranking: cross-encoders and bi-encoders. -NOTE: In this version, {es} *only supports cross-encoders* for semantic reranking. +NOTE: In this version, {es} *only supports cross-encoders* for semantic re-ranking. * A *cross-encoder model* can be thought of as a more powerful, all-in-one solution, because it generates query-aware document representations. It takes the query and document texts as a single, concatenated input. @@ -62,7 +62,7 @@ If you're interested in a more detailed analysis of the practical differences be .Comparisons between cross-encoder and bi-encoder [%collapsible] ============== -The following is a non-exhaustive list of considerations when choosing between cross-encoders and bi-encoders for semantic reranking: +The following is a non-exhaustive list of considerations when choosing between cross-encoders and bi-encoders for semantic re-ranking: * Because a cross-encoder model simultaneously processes both query and document texts, it can better infer their relevance, making it more effective as a reranker than a bi-encoder. * Cross-encoder models are generally larger and more computationally intensive, resulting in higher latencies and increased computational costs. @@ -74,28 +74,28 @@ For example, their ability to take word order into account can improve on dense This enables you to maintain high relevance in result sets, by setting a minimum score threshold for all queries. For example, this is important when using results in a RAG workflow or if you're otherwise feeding results to LLMs. Note that similarity scores from bi-encoders/embedding similarities are _query-dependent_, meaning you cannot set universal cut-offs. -* Bi-encoders rerank using embeddings. You can improve your reranking latency by creating embeddings at ingest-time. These embeddings can be stored for reranking without being indexed for retrieval, reducing your memory footprint. +* Bi-encoders rerank using embeddings. You can improve your re-ranking latency by creating embeddings at ingest-time. These embeddings can be stored for re-ranking without being indexed for retrieval, reducing your memory footprint. ============== [discrete] [[semantic-reranking-in-es]] -=== Semantic reranking in {es} +=== Semantic re-ranking in {es} -In {es}, semantic rerankers are implemented using the {es} <> and a <>. +In {es}, semantic re-rankers are implemented using the {es} <> and a <>. -To use semantic reranking in {es}, you need to: +To use semantic re-ranking in {es}, you need to: -. *Choose a reranking model*. +. *Choose a re-ranking model*. Currently you can: ** Integrate directly with the <> using the `rerank` task type ** Integrate directly with the <> using the `rerank` task type -** Upload a model to {es} from Hugging Face with {eland-docs}/machine-learning.html#ml-nlp-pytorch[Eland]. You'll need to use the `text_similarity` NLP task type when loading the model using Eland. Refer to {ml-docs}/ml-nlp-model-ref.html#ml-nlp-model-ref-text-similarity[the Elastic NLP model reference] for a list of third party text similarity models supported by {es} for semantic reranking. +** Upload a model to {es} from Hugging Face with {eland-docs}/machine-learning.html#ml-nlp-pytorch[Eland]. You'll need to use the `text_similarity` NLP task type when loading the model using Eland. Refer to {ml-docs}/ml-nlp-model-ref.html#ml-nlp-model-ref-text-similarity[the Elastic NLP model reference] for a list of third party text similarity models supported by {es} for semantic re-ranking. *** Then set up an <> with the `rerank` task type . *Create a `rerank` task using the <>*. -The Inference API creates an inference endpoint and configures your chosen machine learning model to perform the reranking task. +The Inference API creates an inference endpoint and configures your chosen machine learning model to perform the re-ranking task. . *Define a `text_similarity_reranker` retriever in your search request*. -The retriever syntax makes it simple to configure both the retrieval and reranking of search results in a single API call. +The retriever syntax makes it simple to configure both the retrieval and re-ranking of search results in a single API call. .*Example search request* with semantic reranker [%collapsible] diff --git a/docs/reference/search/retriever.asciidoc b/docs/reference/search/retriever.asciidoc index 58cc8ce9ef459..6d3a1a36ad407 100644 --- a/docs/reference/search/retriever.asciidoc +++ b/docs/reference/search/retriever.asciidoc @@ -325,7 +325,7 @@ The `text_similarity_reranker` retriever uses an NLP model to improve search res [TIP] ==== -Refer to <> for a high level overview of semantic reranking. +Refer to <> for a high level overview of semantic re-ranking. ==== ===== Prerequisites @@ -387,7 +387,7 @@ A text similarity re-ranker retriever is a compound retriever. Child retrievers [[text-similarity-reranker-retriever-example-cohere]] ==== Example: Cohere Rerank -This example enables out-of-the-box semantic search by reranking top documents using the Cohere Rerank API. This approach eliminate the need to generate and store embeddings for all indexed documents. +This example enables out-of-the-box semantic search by re-ranking top documents using the Cohere Rerank API. This approach eliminate the need to generate and store embeddings for all indexed documents. This requires a <> using the `rerank` task type. [source,js] @@ -418,7 +418,7 @@ GET /index/_search [discrete] [[text-similarity-reranker-retriever-example-eland]] -==== Example: Semantic reranking with a Hugging Face model +==== Example: Semantic re-ranking with a Hugging Face model The following example uses the `cross-encoder/ms-marco-MiniLM-L-6-v2` model from Hugging Face to rerank search results based on semantic similarity. The model must be uploaded to {es} using https://www.elastic.co/guide/en/elasticsearch/client/eland/current/machine-learning.html#ml-nlp-pytorch[Eland]. @@ -428,7 +428,7 @@ The model must be uploaded to {es} using https://www.elastic.co/guide/en/elastic Refer to {ml-docs}/ml-nlp-model-ref.html#ml-nlp-model-ref-text-similarity[the Elastic NLP model reference] for a list of third party text similarity models supported by {es}. ==== -Follow these steps to load the model and create a semantic reranker. +Follow these steps to load the model and create a semantic re-ranker. . Install Eland using `pip` + From 9f4d508ba708bbb61c1df25aee5cc28cc801706d Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Thu, 19 Sep 2024 15:47:56 +0200 Subject: [PATCH 09/12] Fix semantic text verbiage Co-authored-by: Kathleen DeRusso --- docs/reference/reranking/index.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/reranking/index.asciidoc b/docs/reference/reranking/index.asciidoc index d501b2c038f62..518257dcf7a86 100644 --- a/docs/reference/reranking/index.asciidoc +++ b/docs/reference/reranking/index.asciidoc @@ -32,7 +32,7 @@ https://en.wikipedia.org/wiki/Okapi_BM25[BM25] is the default statistical scorin ==== Vector search: similarity scoring Vector search involves transforming data into dense or sparse vector embeddings to capture semantic meanings, and computing similarity scores for query vectors. -Store vectors using `semantic` fields for automatic vectorization or `dense_vector` and `sparse_vector` fields when you need more control over the underlying embedding model. +Store vectors using `semantic_text` fields for automatic inference and vectorization or `dense_vector` and `sparse_vector` fields when you need more control over the underlying embedding model. Query vectors with `semantic` queries or `knn` and `sparse_vector` queries to compute similarity scores. Refer to <> for more information. From a351297455c4d0203e5f87249cf850c710ce27ff Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Thu, 19 Sep 2024 15:48:20 +0200 Subject: [PATCH 10/12] Fix vector query verbiage Co-authored-by: Kathleen DeRusso --- docs/reference/reranking/index.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/reranking/index.asciidoc b/docs/reference/reranking/index.asciidoc index 518257dcf7a86..d38c24dd72dfc 100644 --- a/docs/reference/reranking/index.asciidoc +++ b/docs/reference/reranking/index.asciidoc @@ -33,7 +33,7 @@ https://en.wikipedia.org/wiki/Okapi_BM25[BM25] is the default statistical scorin Vector search involves transforming data into dense or sparse vector embeddings to capture semantic meanings, and computing similarity scores for query vectors. Store vectors using `semantic_text` fields for automatic inference and vectorization or `dense_vector` and `sparse_vector` fields when you need more control over the underlying embedding model. -Query vectors with `semantic` queries or `knn` and `sparse_vector` queries to compute similarity scores. +Query vector fields with `semantic`, `knn` or `sparse_vector` queries to compute similarity scores. Refer to <> for more information. [float] From babc1a0b2183461ef005a9ff0d5990ce3ac08a9d Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Thu, 19 Sep 2024 16:48:25 +0200 Subject: [PATCH 11/12] Get back in the box Co-authored-by: Kathleen DeRusso --- docs/reference/reranking/index.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/reranking/index.asciidoc b/docs/reference/reranking/index.asciidoc index d38c24dd72dfc..f1ac669a80bb4 100644 --- a/docs/reference/reranking/index.asciidoc +++ b/docs/reference/reranking/index.asciidoc @@ -25,7 +25,7 @@ Because the resource-intensive model is only applied to the smaller set of pre-f ==== Full-text search: BM25 scoring {es} ranks documents based on term frequency and inverse document frequency, adjusted for document length. -https://en.wikipedia.org/wiki/Okapi_BM25[BM25] is the default statistical scoring algorithm in {es} and works out-of-the-box. +https://en.wikipedia.org/wiki/Okapi_BM25[BM25] is the default statistical scoring algorithm in {es}. [float] [[re-ranking-ranking-overview-vector]] From ef7687481dbf3979ad8c31afc58cf3812b349063 Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Thu, 19 Sep 2024 16:50:10 +0200 Subject: [PATCH 12/12] sentence case --- docs/reference/reranking/index.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/reranking/index.asciidoc b/docs/reference/reranking/index.asciidoc index f1ac669a80bb4..cc6f4a9007424 100644 --- a/docs/reference/reranking/index.asciidoc +++ b/docs/reference/reranking/index.asciidoc @@ -45,7 +45,7 @@ Hybrid search techniques combine results from full-text and vector search pipeli [float] [[re-ranking-overview-second-stage]] -=== Second Stage: Re-ranking +=== Second stage: Re-ranking When using the following advanced re-ranking pipelines, first-stage retrieval mechanisms effectively generate a set of candidates. These candidates are funneled into the re-ranker to perform more computationally expensive re-ranking tasks.