Provide more details about vectorization

Author: lcawl

solutions/search/get-started/semantic-search.md

@@ -13,68 +13,65 @@ _Semantic search_ is a type of AI-powered search that enables you to use natural
 It returns results that match the meaning of a query, as opposed to literal keyword matches.
 For example, if you want to search for workplace guidelines on a second income, you could search for "side hustle", which is not a term you're likely to see in a formal HR document.
 
+Semantic search uses {{es}} vector database and vector search technology.
+Each _vector_ (or _vector embedding_) is an array of numbers that each represent a different characteristic of the text, such as sentiment, context, and syntactics.
+These numeric representations make comparison with other vectors very efficient.
+
 In this guide, you'll learn how to perform semantic search on a small set of sample data.
-You'll use the default Learned Sparse Encoder model ([ELSER](/explore-analyze/machine-learning/nlp/ml-nlp-elser.md)), which automatically creates vector embeddings (numeric representations that capture the text meaning) when storing and searching the data.
-This model is built to provide great relevance across domains, without the need for additional fine tuning.
+You'll create vectors and store them in {{es}}.
+Then you'll run a query, which will be transformed into vectors and compared to the stored data.
+By playing with a simple use case, you'll take the first steps toward understanding whether this type of search is relevant to your own data.
 
 ## Prerequisites
 
 - If you're using [{{es-serverless}}](/solutions/search/serverless-elasticsearch-get-started.md), create a project that is optimized for vectors. To add the sample data, you must have a `developer` or `admin` predefined role or an equivalent custom role.
-- If you're [running {{es}} locally](/solutions/search/run-elasticsearch-locally.md), start {{es}} and {{kib}}. To add the sample data, log in with the `elastic` user that has the `superuser` built-in role.
+- If you're [running {{es}} locally](/solutions/search/run-elasticsearch-locally.md), start {{es}} and {{kib}}. To add the sample data, log in with a user that has the `superuser` built-in role, such as `elastic`.
 
 To learn about role-based access control, check out [](/deploy-manage/users-roles/cluster-or-deployment-auth/user-roles.md).
 
 <!--
-TBD: It seems like semantic search fields exist in all, so what is the value of this "optimized for vectors" option?
+TBD: What is the impact of this "optimized for vectors" option?
 -->
 
-## Add data
+## Create a vector database
 
-% TBD: What type of data is ideal for semantic search?
+When you create vectors (or _vectorize_ your data), you convert complex and nuanced documents into multidimensional numerical representations.
+You can choose from many different vector embedding models. Some are extremely hardware efficient and can be run with less computational power. Others have a greater “understanding” of the context and can answer questions and lead a threaded conversation.
+These examples use the default Learned Sparse Encoder ([ELSER](/explore-analyze/machine-learning/nlp/ml-nlp-elser.md)) model, which provides great relevance across domains without the need for additional fine tuning.
 
-::::{tab-set}
-:::{tab-item} {{serverless-short}}
-:sync: serverless
-There are some small data sets available for learning purposes.
-Go to **{{es}} > Home**, select the semantic search workflow, and click **Create a semantic optimized index**.
+The way that you store and index vectors has a significant impact on the performance and accuracy of search results.
+They must be stored in specialized data structures designed to ensure efficient similarity search and speedy vector distance calculations.
+These examples store the vectors in `semantic_text` fields, which provide sensible defaults and automation.
 
-Follow the instructions to install an {{es}} client and copy the code examples.
-Alternatively, try out the API requests in the [Console](/explore-analyze/query-filter/tools/console.md).
-:::
-:::{tab-item} {{stack}}
-:sync: stack
-There are some small data sets available for learning purposes.
-Go to **{{es}} > Home** and click **Create API index**.
-Select the semantic search workflow in the guided index flow.
-
-Follow the instructions to install an {{es}} client and copy the code examples.
-Alternatively, try out the API requests in the [Console](/explore-analyze/query-filter/tools/console.md).
-:::
-::::
+Try vectorizing a small set of documents.
+You can follow the guided index workflow:
 
-:::::{stepper}
+- If you're using {{es-serverless}}, go to **{{es}} > Home**, select the semantic search workflow, and click **Create a semantic optimized index**.
+- If you're running {{es}} locally, go to **{{es}} > Home** and click **Create API index**. Select the semantic search workflow.
 
-::::{step} Create a `semantic_text` field mapping
+Alternatively, run the following API requests in the [Console](/explore-analyze/query-filter/tools/console.md):
 
-You can implement semantic search with varying levels of complexity and customization.
-The recommended method is to use `semantic_text` fields, which provide sensible defaults and automation.
-For example, it uses [ELSER](/explore-analyze/machine-learning/nlp/ml-nlp-elser.md) (and therefore sparse vectors) by default.
+:::::{stepper}
+::::{step} Create a semantic_text field mapping
 
-The following example creates a mapping for a single field:
+The following example creates a mapping for a single field ("content"):
 
 ```console
 PUT /semantic-index/_mapping
 {
   "properties": {
-    "text": {
+    "content": {
       "type": "semantic_text"
     }
   }
 }
 ```
 
-Refer to [](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md) for details about this field type.
-For a broader overview, check out [Mapping embeddings to Elasticsearch field types: semantic_text, dense_vector, sparse_vector](https://www.elastic.co/search-labs/blog/mapping-embeddings-to-elasticsearch-field-types).
+When you use `semantic_text` fields, the type of vector is determined by the vector embedding model.
+In this case, the default ELSER model will be used to create sparse vectors.
+
+For more details about `semantic_text` fields, refer to [](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md).
+For a deeper dive, check out [Mapping embeddings to Elasticsearch field types: semantic_text, dense_vector, sparse_vector](https://www.elastic.co/search-labs/blog/mapping-embeddings-to-elasticsearch-field-types).
 ::::
 
 ::::{step} Add documents
@@ -84,30 +81,26 @@ You can use the Elasticsearch bulk API to ingest an array of documents:
 ```console
 POST /_bulk?pretty
 { "index": { "_index": "semantic-index" } }
-{"text":"Yellowstone National Park is one of the largest national parks in the United States. It ranges from the Wyoming to Montana and Idaho, and contains an area of 2,219,791 acress across three different states. Its most famous for hosting the geyser Old Faithful and is centered on the Yellowstone Caldera, the largest super volcano on the American continent. Yellowstone is host to hundreds of species of animal, many of which are endangered or threatened. Most notably, it contains free-ranging herds of bison and elk, alongside bears, cougars and wolves. The national park receives over 4.5 million visitors annually and is a UNESCO World Heritage Site."}
+{"content":"Yellowstone National Park is one of the largest national parks in the United States. It ranges from the Wyoming to Montana and Idaho, and contains an area of 2,219,791 acress across three different states. Its most famous for hosting the geyser Old Faithful and is centered on the Yellowstone Caldera, the largest super volcano on the American continent. Yellowstone is host to hundreds of species of animal, many of which are endangered or threatened. Most notably, it contains free-ranging herds of bison and elk, alongside bears, cougars and wolves. The national park receives over 4.5 million visitors annually and is a UNESCO World Heritage Site."}
 { "index": { "_index": "semantic-index" } }
-{"text":"Yosemite National Park is a United States National Park, covering over 750,000 acres of land in California. A UNESCO World Heritage Site, the park is best known for its granite cliffs, waterfalls and giant sequoia trees. Yosemite hosts over four million visitors in most years, with a peak of five million visitors in 2016. The park is home to a diverse range of wildlife, including mule deer, black bears, and the endangered Sierra Nevada bighorn sheep. The park has 1,200 square miles of wilderness, and is a popular destination for rock climbers, with over 3,000 feet of vertical granite to climb. Its most famous and cliff is the El Capitan, a 3,000 feet monolith along its tallest face."}
+{"content":"Yosemite National Park is a United States National Park, covering over 750,000 acres of land in California. A UNESCO World Heritage Site, the park is best known for its granite cliffs, waterfalls and giant sequoia trees. Yosemite hosts over four million visitors in most years, with a peak of five million visitors in 2016. The park is home to a diverse range of wildlife, including mule deer, black bears, and the endangered Sierra Nevada bighorn sheep. The park has 1,200 square miles of wilderness, and is a popular destination for rock climbers, with over 3,000 feet of vertical granite to climb. Its most famous and cliff is the El Capitan, a 3,000 feet monolith along its tallest face."}
 { "index": { "_index": "semantic-index" } }
-{"text":"Rocky Mountain National Park  is one of the most popular national parks in the United States. It receives over 4.5 million visitors annually, and is known for its mountainous terrain, including Longs Peak, which is the highest peak in the park. The park is home to a variety of wildlife, including elk, mule deer, moose, and bighorn sheep. The park is also home to a variety of ecosystems, including montane, subalpine, and alpine tundra. The park is a popular destination for hiking, camping, and wildlife viewing, and is a UNESCO World Heritage Site."}
+{"content":"Rocky Mountain National Park  is one of the most popular national parks in the United States. It receives over 4.5 million visitors annually, and is known for its mountainous terrain, including Longs Peak, which is the highest peak in the park. The park is home to a variety of wildlife, including elk, mule deer, moose, and bighorn sheep. The park is also home to a variety of ecosystems, including montane, subalpine, and alpine tundra. The park is a popular destination for hiking, camping, and wildlife viewing, and is a UNESCO World Heritage Site."}
 ```
 
 The bulk ingestion request might take longer than the default request timeout.
 If it times out, wait for the machine learning model loading to complete (typically 1-5 minutes) then retry it.
-
-<!--
-TBD: Describe where to look for the downloaded model in Trained Models?
--->
+::::
+:::::
 
 What just happened? The content was transformed into a sparse vector, which involves two main steps.
 First, the content is divided into smaller, manageable chunks to ensure that meaningful segments can be more effectively processed and searched. Then each chunk of text is transformed into a sparse vector representation using text expansion techniques.
-By default, `semantic_text` fields leverage ELSER to transform the content.
-
-% TBD: Confirm "Elser model" vs ".elser-2-elasticsearch" terminology.
 
 ![Semantic search chunking](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9bbe5e260012b15d/67ffffc8165067d96124b586/animated-gif-semantic-search-chunking.gif)
 
-::::
-::::{step} Explore the data
+
+## Explore the data
+
 To familiarize yourself with this data set, open [Discover](/explore-analyze/discover.md) from the navigation menu or by using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
 
 In **Discover**, you can click the expand icon ![double arrow icon to open a flyout with the document details](/explore-analyze/images/kibana-expand-icon-2.png "") to show details about any documents in the table.
@@ -118,15 +111,19 @@ In **Discover**, you can click the expand icon ![double arrow icon to open a fly
 :::
 
 For more tips, check out [](/explore-analyze/discover/discover-get-started.md).
-::::
-:::::
+
 <!--
 TBD: When you view these documents in Discover they're shown as having "text" field type instead of "semantic_text" is this right?
-TBD: Should we call out that the KQL filters in Discover don't seem to work against semantic_text fields yet?
 -->
 
 ## Test semantic search
 
+<!--
+TO-DO: Talk about the pipeline where vectors are required for both the data and search query
+% encodes details of searchable information into vectors and then compares vectors to determine which are most similar.
+When you run a query, the search engine transforms the query into embeddings, which are numerical representations of data and related contexts. They are stored in vectors. The kNN algorithm, or k-nearest neighbor algorithm, then matches vectors of existing documents (a semantic search concerns text) to the query vectors. The semantic search then generates results and ranks them based on conceptual relevance.
+-->
+
 {{es}} provides a variety of query languages for interacting with your data.
 For an overview of their features and use cases, check out [](/explore-analyze/query-filter/languages.md).
 
@@ -155,7 +152,7 @@ POST /semantic-index/_search
     "standard": {
       "query": {
         "semantic": {
-          "field": "text",
+          "field": "content",
           "query": "best park for rappelling"
         }
       }
@@ -191,7 +188,7 @@ The search results are sorted by a relevance score, which measures how well each
         "_id": "Pp0MtJcBZjjo1YKoXkWH",
         "_score": 11.389743,
         "_source": {
-          "text": "Rocky Mountain National Park ..."
+          "content": "Rocky Mountain National Park ..."
   ...
 }
 ```
@@ -213,8 +210,8 @@ Copy the following query:
 
 ```esql
 FROM semantic-index METADATA _score <1>
-  | WHERE text: "what's the biggest park?" <2>
-  | KEEP text, _score <3>
+  | WHERE content: "what's the biggest park?" <2>
+  | KEEP content, _score <3>
   | SORT _score DESC <4>
   | LIMIT 1000 <5>
 ```