[Search] Vector & semantic search

Author: leemthompo

images/elasticsearch-reference-semantic-options.svg

@@ -5,11 +5,8 @@ mapped_pages:
 
 # Full-text search [full-text-search]
 
-::::{admonition} Hands-on introduction to full-text search
-:class: tip
-
-Would you prefer to jump straight into a hands-on tutorial? Refer to our quick start [full-text search tutorial](get-started.md).
-
+::::{tip}
+Would you prefer to start with a hands-on example? Refer to our [full-text search tutorial](querydsl-full-text-filter-tutorial.md).
 ::::
 
 Full-text search, also known as lexical search, is a technique for fast, efficient searching through text fields in documents. Documents and search queries are transformed to enable returning [relevant](https://www.elastic.co/what-is/search-relevance) results instead of simply exact term matches. Fields of type [`text`](https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html#text-field-type) are analyzed and indexed for full-text search.

raw-migrated-files/elasticsearch/elasticsearch-reference/semantic-search.md

@@ -1,26 +1,78 @@
----
-mapped_urls:
-  - https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-search.html
-  - https://www.elastic.co/guide/en/serverless/current/elasticsearch-reference-semantic-search.html
----
+# Semantic search [semantic-search]
 
-# Semantic search
+:::{note}
+This page focuses on the semantic search workflows available in {{es}}. For detailed information about vector search implementations, refer to [vector search](vector.md).
+:::
 
-% What needs to be done: Lift-and-shift
+Sometimes [full-text search](full-text.md) isn't enough.
 
-% Use migrated content from existing pages that map to this page:
+Semantic search techniques help users find data based on intent and contextual meaning, going beyond traditional keyword matching.
 
-% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/semantic-search.md
-% - [ ] ./raw-migrated-files/docs-content/serverless/elasticsearch-reference-semantic-search.md
+Semantic search has a wide range of use cases, including:
 
-% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):
+- **Question answering**: Find the most relevant answers to user questions
+- **Recommendation systems**: Suggest similar items based on user preferences
+- **Information retrieval**: Retrieve relevant information from large datasets
+- **Product discovery**: Help E-commerce users find relevant products
+- **Workplace search**: Help employees find relevant information within an organization
 
-$$$elasticsearch-reference-semantic-search-semantic-text$$$
+{{es}} provides various semantic search capabilities using [natural language processing (NLP)](../../raw-migrated-files/elasticsearch/explore-analyze/machine-learning/nlp.md) and [vector search](vector.md).
 
-$$$elasticsearch-reference-semantic-search-inference-api$$$
+## Overview of semantic search workflows [semantic-search-workflows-overview]
 
-$$$elasticsearch-reference-semantic-search-model-deployment$$$
+You have several options for using NLP models for semantic search in the {{stack}}:
 
-$$$using-nlp-models$$$
+* Option 1: Use the `semantic_text` workflow (recommended)
+* Option 2: Use the {{infer}} API workflow
+* Option 3: Deploy models directly in {{es}}
 
-$$$using-query$$$
+This diagram summarizes the relative complexity of each workflow:
+
+:::{image} ../../../images/elasticsearch-reference-semantic-options.svg
+:alt: Overview of semantic search workflows in {{es}}
+:::
+
+## Choose a semantic search workflow [using-nlp-models]
+
+### Option 1: `semantic_text` [_semantic_text_workflow]
+
+The simplest way to use NLP models in the {{stack}} is through the [`semantic_text` workflow](semantic-search/semantic-search-semantic-text.md). We recommend using this approach because it abstracts away a lot of manual work. All you need to do is create an {{infer}} endpoint and an index mapping to start ingesting, embedding, and querying data. There is no need to define model-related settings and parameters, or to create {{infer}} ingest pipelines. Refer to the [Create an {{infer}} endpoint API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html) documentation for a list of supported services.
+
+For an end-to-end tutorial, refer to [Semantic search with `semantic_text`](semantic-search/semantic-search-semantic-text.md).
+
+
+### Option 2: Inference API [_infer_api_workflow]
+
+The [{{infer}} API workflow](inference-api.md) is more complex but offers greater control over the {{infer}} endpoint configuration. You need to create an {{infer}} endpoint, provide various model-related settings and parameters, define an index mapping, and set up an {{infer}} ingest pipeline with the appropriate settings.
+
+For an end-to-end tutorial, refer to [Semantic search with the {{infer}} API](inference-api.md).
+
+
+### Option 3: Manual model deployment [_model_deployment_workflow]
+
+You can also deploy NLP in {{es}} manually, without using an {{infer}} endpoint. This is the most complex and labor intensive workflow for performing semantic search in the {{stack}}. You need to select an NLP model from the [list of supported dense and sparse vector models](../../explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md#ml-nlp-model-ref-text-embedding), deploy it using the Eland client, create an index mapping, and set up a suitable ingest pipeline to start ingesting and querying data.
+
+For an end-to-end tutorial, refer to [Semantic search with a model deployed in {{es}}](semantic-search/semantic-search-deployed-nlp-model.md).
+
+::::{tip}
+Refer to [vector queries and field types](vector.md#choosing-vector-query) for a quick reference overview.
+::::
+
+## Learn more [semantic-search-read-more]
+
+### Interactive examples
+
+- The [`elasticsearch-labs`](https://github.com/elastic/elasticsearch-labs) repo contains a number of interactive semantic search examples in the form of executable Python notebooks, using the {{es}} Python client
+- [Semantic search with ELSER using the model deployment workflow](https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/03-ELSER.ipynb)
+- [Semantic search with `semantic_text`](https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/09-semantic-text.ipynb)
+
+### Blogs
+
+- [{{es}} new semantic_text mapping: Simplifying semantic search](https://www.elastic.co/search-labs/blog/semantic-search-simplified-semantic-text)
+- [Introducing Elastic Learned Sparse Encoder: Elastic's AI model for semantic search](https://www.elastic.co/blog/may-2023-launch-sparse-encoder-ai-model)
+- [How to get the best of lexical and AI-powered search with Elastic's vector database](https://www.elastic.co/blog/lexical-ai-powered-search-elastic-vector-database)
+- Information retrieval blog series:
+    - [Part 1: Steps to improve search relevance](https://www.elastic.co/blog/improving-information-retrieval-elastic-stack-search-relevance)
+    - [Part 2: Benchmarking passage retrieval](https://www.elastic.co/blog/improving-information-retrieval-elastic-stack-benchmarking-passage-retrieval)
+    - [Part 3: Introducing Elastic Learned Sparse Encoder, our new retrieval model](https://www.elastic.co/blog/may-2023-launch-information-retrieval-elasticsearch-ai-model)
+    - [Part 4: Hybrid retrieval](https://www.elastic.co/blog/improving-information-retrieval-elastic-stack-hybrid)