Adding more retriever examples to the documentation examples

docs/reference/search/search-your-data/retrievers-overview.asciidoc

@@ -67,59 +67,131 @@ When using compound retrievers, only the query element is allowed, which enforce
 
 [discrete]
 [[retrievers-overview-example]]
-==== Example
+==== Examples
 
-The following example demonstrates the powerful queries that we can now compose, and how retrievers simplify this process. We can use any combination of retrievers we want, propagating the
-results of a nested retriever to its parent. In this scenario, we'll make use of all 4 (currently) available retrievers, i.e. `standard`, `knn`, `text_similarity_reranker` and `rrf`.
-We'll first combine the results of a `semantic` query using the `standard` retriever, and that of a `knn` search on a dense vector field, using `rrf` to get the top 100 results.
-Finally, we'll then rerank the top-50 results of `rrf` using the `text_similarity_reranker`
+Let's work through some examples to see how we can use and leverage retrievers to build an awesome search experience!
+
+To show the full functionality, in this exercise, we'll assume that we have access to a reranker model through the inference service,
+as well as access to Elser, for building semantic queries!
+
+So, first things first, let's start by setting up these services and have them in place for later use!
+
+[source,js]
+----
+// Setup rerank task stored as `my-awesome-rerank-model`
+PUT _inference/rerank/my-awesome-rerank-model
+{
+ "service": "cohere",
+ "service_settings": {
+   "model_id": "rerank-english-v3.0",
+   "api_key": "{{COHERE_API_KEY}}"
+ }
+}
+----
+//NOTCONSOLE
+
+[source,js]
+----
+// Setup ELSER as `my-elser-endpoint`
+PUT _inference/sparse_embedding/my-elser-endpoint
+{
+ "service": "elser",
+ "service_settings": {
+   "num_allocations": 1,
+   "num_threads": 1
+ },
+ "task_settings": {}
+}
+----
+//NOTCONSOLE
+
+Now that we have our services in place, lets create our `retrievers_example` index, and add some documents to it.
+[source,js]
+----
+PUT retrievers_example
+{
+   "mappings": {
+       "properties": {
+           "vector": {
+               "type": "dense_vector",
+               "dims": 3,
+               "similarity": "l2_norm",
+               "index": true
+           },
+           "text": {
+               "type": "text",
+               "copy_to": "inference_field"
+           },
+           "year": {
+               "type": "integer"
+           },
+           "topic": {
+               "type": "keyword"
+           },
+           "inference_field": {
+               "type": "semantic_text",
+               "inference_id": "my-elser-endpoint"
+           }
+       }
+   }
+}
+----
+//NOTCONSOLE
 
 [source,js]
 ----
-GET example-index/_search
+POST /retrievers_example/_doc/1
+{
+ "vector": [0.23, 0.67, 0.89],
+ "text": "Large language models are revolutionizing information retrieval by boosting search precision, deepening contextual understanding, and reshaping user experiences in data-rich environments.",
+ "year": 2024,
+ "topic": ["llm", "ai", "information_retrieval"]
+}
+
+POST /retrievers_example/_doc/2
+{
+ "vector": [0.12, 0.56, 0.78],
+ "text": "Artificial intelligence is transforming medicine, from advancing diagnostics and tailoring treatment plans to empowering predictive patient care for improved health outcomes.",
+ "year": 2023,
+ "topic": ["ai", "medicine"]
+}
+
+POST /retrievers_example/_doc/3
 {
-  "retriever": {
-    "text_similarity_reranker": {
-      "retriever": {
-        "rrf": {
-          "retrievers": [
-            {
-                "standard": {
-                  "query": {
-                    "semantic": {
-                      "field": "inference_field",
-                      "query": "state of the art vector database"
-                    }
-                  }
-                }
-            },
-            {
-              "knn": {
-                "query_vector": [
-                  0.54,
-                  ...,
-                  0.245
-                ],
-                "field": "embedding",
-                "k": 10,
-                "num_candidates": 15
-              }
-            }
-          ],
-          "rank_window_size": 100,
-          "rank_constant": 10
-        }
-      },
-      "rank_window_size": 50,
-      "field": "description",
-      "inference_text": "what's the best way to create complex pipelines and retrieve documents?",
-      "inference_id": "my-awesome-rerank-model"
-    }
-  }
+ "vector": [0.45, 0.32, 0.91],
+  "text": "AI is redefining security by enabling advanced threat detection, proactive risk analysis, and dynamic defenses against increasingly sophisticated cyber threats.",
+ "year": 2024,
+ "topic": ["ai", "security"]
 }
+
+POST /retrievers_example/_doc/4
+{
+ "vector": [0.34, 0.21, 0.98],
+ "text": "Elastic introduces Elastic AI Assistant, the open, generative AI sidekick powered by ESRE to democratize cybersecurity and enable users of every skill level.",
+ "year": 2023,
+ "topic": ["ai", "elastic", "assistant"]
+}
+
+POST /retrievers_example/_doc/5
+{
+ "vector": [0.11, 0.65, 0.47],
+ "text": "Learn how to spin up a deployment of our hosted Elasticsearch Service and use Elastic Observability to gain deeper insight into the behavior of your applications and systems.",
+ "year": 2024,
+ "topic": ["documentation", "observability", "elastic"]
+}
+
 ----
 //NOTCONSOLE
 
+Now that we also have our documents in place, let's try to run some queries using retrievers!
+
+include::retrievers_examples.asciidoc[tag=basic-rrf-retriever-with-semantic-query]
+include::retrievers_examples.asciidoc[tag=rrf-retriever-with-collapse]
+include::retrievers_examples.asciidoc[tag=rrf-on-top-of-semantic-reranker]
+include::retrievers_examples.asciidoc[tag=text-similarity-reranker-on-top-of-rrf]
+include::retrievers_examples.asciidoc[tag=chaining-text-similarity-reranker-retrievers]
+include::retrievers_examples.asciidoc[tag=rrf-retriever-with-aggs]
+
 [discrete]
 [[retrievers-overview-glossary]]
 ==== Glossary