Skip to content

Commit e0d6769

Browse files
szabostevekosabogi
andauthored
[SEARCH] Improves sparse vector and vector search tutorial pages based on SEO guidelines (#2793)
## Description This PR optimizes the content of the `Sparse vector` and `Vector search tutorial` pages based on the SEO best practices outlined [here](https://stunning-adventure-qrvr1k2.pages.github.io/style-guide/seo/). --------- Co-authored-by: kosabogi <[email protected]>
1 parent d1f956d commit e0d6769

File tree

2 files changed

+32
-34
lines changed

2 files changed

+32
-34
lines changed

solutions/search/vector/dense-versus-sparse-ingest-pipelines.md

Lines changed: 25 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -11,31 +11,27 @@ products:
1111

1212
# Tutorial: Dense and sparse workflows using ingest pipelines [semantic-search-deployed-nlp-model]
1313

14-
1514
::::{important}
16-
* For the easiest way to perform semantic search in the {{stack}}, refer to the [`semantic_text`](../semantic-search/semantic-search-semantic-text.md) end-to-end tutorial.
17-
* This tutorial was written before the [{{infer}} endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-inference) and [`semantic_text` field type](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md) was introduced. Today we have simpler options for performing semantic search.
1815

16+
* For the easiest way to perform semantic search in the {{stack}}, refer to the [`semantic_text`](../semantic-search/semantic-search-semantic-text.md) end-to-end tutorial.
17+
* This tutorial predates the [{{infer}} endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-inference) and the [`semantic_text` field type](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md). Today there are simpler, higher-level options for semantic search than the ones outlined in this tutorial. The semantic text workflow is the recommended way to perform semantic search for most use cases.
1918
::::
2019

21-
22-
This guide shows you how to implement semantic search with models deployed in {{es}}: from selecting an NLP model, to writing queries.
23-
20+
**This guide shows how to implement semantic search in {{es}} with deployed NLP models: from selecting a model, to configuring ingest pipelines, to running queries.**
2421

2522
## Select an NLP model [deployed-select-nlp-model]
2623

27-
{{es}} offers the usage of a [wide range of NLP models](/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md#ml-nlp-model-ref-text-embedding), including both dense and sparse vector models. Your choice of the language model is critical for implementing semantic search successfully.
28-
29-
While it is possible to bring your own text embedding model, achieving good search results through model tuning is challenging. Selecting an appropriate model from our third-party model list is the first step. Training the model on your own data is essential to ensure better search results than using only BM25. However, the model training process requires a team of data scientists and ML experts, making it expensive and time-consuming.
24+
{{es}} supports a [wide range of NLP models](/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md#ml-nlp-model-ref-text-embedding), including **dense** and **sparse** vector models. Choosing the right model is critical for good relevance.
3025

31-
To address this issue, Elastic provides a pre-trained representational model called [Elastic Learned Sparse EncodeR (ELSER)](/explore-analyze/machine-learning/nlp/ml-nlp-elser.md). ELSER, currently available only for English, is an out-of-domain sparse vector model that does not require fine-tuning. This adaptability makes it suitable for various NLP use cases out of the box. Unless you have a team of ML specialists, it is highly recommended to use the ELSER model.
26+
While bringing your own text embedding model is possible, achieving strong results typically requires tuning and evaluation. Selecting a well-performing third-party model from our list is a good start. Training the model on your own data is essential to ensure better search results than using only BM25. However, the model training process requires a team of data scientists and ML experts, making it expensive and time-consuming.
3227

33-
In the case of sparse vector representation, the vectors mostly consist of zero values, with only a small subset containing non-zero values. This representation is commonly used for textual data. In the case of ELSER, each document in an index and the query text itself are represented by high-dimensional sparse vectors. Each non-zero element of the vector corresponds to a term in the model vocabulary. The ELSER vocabulary contains around 30000 terms, so the sparse vectors created by ELSER contain about 30000 values, the majority of which are zero. Effectively the ELSER model is replacing the terms in the original query with other terms that have been learnt to exist in the documents that best match the original search terms in a training dataset, and weights to control how important each is.
28+
To lower the barrier, Elastic provides [**Elastic Learned Sparse EncodeR (ELSER)**](/explore-analyze/machine-learning/nlp/ml-nlp-elser.md), a pre-trained sparse vector model (English-only) that delivers strong out-of-the-box relevance without fine-tuning. This adaptability makes it suitable for various NLP use cases out of the box. Unless you have an ML team, we recommend starting with ELSER.
3429

30+
Sparse vectors are mostly zeros with a small number of non-zero values. This representation is commonly used for textual data. With ELSER, both documents and queries are represented as high-dimensional sparse vectors. Each non-zero element of the vector corresponds to a term in the model vocabulary. The ELSER vocabulary contains around 30000 terms, so the sparse vectors created by ELSER contain about 30000 values, the majority of which are zero. The ELSER model replaces the terms in the original query with other terms learned from the training dataset that appear in documents matching the original search terms, and assigns weights to control their importance.
3531

36-
## Deploy the model [deployed-deploy-nlp-model]
32+
## Deploy the model in {{es}} [deployed-deploy-nlp-model]
3733

38-
After you decide which model you want to use for implementing semantic search, you need to deploy the model in {{es}}.
34+
After you choose a model, deploy it in {{es}}.
3935

4036
:::::::{tab-set}
4137

@@ -51,14 +47,15 @@ To deploy a third-party text embedding model, refer to [Deploy a text embedding
5147

5248
## Map a field for the text embeddings [deployed-field-mappings]
5349

54-
Before you start using the deployed model to generate embeddings based on your input text, you need to prepare your index mapping first. The mapping of the index depends on the type of model.
50+
Before generating embeddings, prepare your **index mapping**. The mapping depends on whether you use a **sparse** (ELSER) or **dense** model.
5551

5652
:::::::{tab-set}
5753

5854
::::::{tab-item} ELSER
59-
ELSER produces token-weight pairs as output from the input text and the query. The {{es}} [`sparse_vector`](elasticsearch://reference/elasticsearch/mapping-reference/sparse-vector.md) field type can store these token-weight pairs as numeric feature vectors. The index must have a field with the `sparse_vector` field type to index the tokens that ELSER generates.
55+
ELSER outputs token-weight pairs. Use the [`sparse_vector`](elasticsearch://reference/elasticsearch/mapping-reference/sparse-vector.md) field to store them.
6056

61-
To create a mapping for your ELSER index, refer to the [Create the index mapping section](../semantic-search/semantic-search-elser-ingest-pipelines.md#elser-mappings) of the tutorial. The example shows how to create an index mapping for `my-index` that defines the `my_embeddings.tokens` field - which will contain the ELSER output - as a `sparse_vector` field.
57+
To create a mapping for your ELSER index, refer to the [Create the index mapping section](../semantic-search/semantic-search-elser-ingest-pipelines.md#elser-mappings) of the tutorial.
58+
Create a mapping with a `sparse_vector` field for the tokens and a `text` field for the source content. For example:
6259

6360
```console
6461
PUT my-index
@@ -76,16 +73,16 @@ PUT my-index
7673
}
7774
```
7875

79-
1. The name of the field that will contain the tokens generated by ELSER.
76+
1. The field that will contain ELSER-generated tokens.
8077
2. The field that contains the tokens must be a `sparse_vector` field.
81-
3. The name of the field from which to create the sparse vector representation. In this example, the name of the field is `my_text_field`.
78+
3. The source field used to create the sparse vector representation. In this example, the name of the field is `my_text_field`.
8279
4. The field type is `text` in this example.
8380
::::::
8481

8582
::::::{tab-item} Dense vector models
86-
The models compatible with {{es}} NLP generate dense vectors as output. The [`dense_vector`](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md) field type is suitable for storing dense vectors of numeric values. The index must have a field with the `dense_vector` field type to index the embeddings that the supported third-party model that you selected generates. Keep in mind that the model produces embeddings with a certain number of dimensions. The `dense_vector` field must be configured with the same number of dimensions using the `dims` option. Refer to the respective model documentation to get information about the number of dimensions of the embeddings.
83+
Dense vector models output numeric embeddings. Use the [`dense_vector`](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md) field type for storing dense vectors that the supported third-party model you selected generates. Keep in mind that the model produces embeddings with a certain number of dimensions. The `dense_vector` field must be configured with the same number of dimensions using the `dims` option. Refer to the respective model documentation to get information about the number of dimensions of the embeddings.
8784

88-
To review a mapping of an index for an NLP model, refer to the mapping code snippet in the [Add the text embedding model to an ingest inference pipeline](/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md#ex-text-emb-ingest) section of the tutorial. The example shows how to create an index mapping that defines the `my_embeddings.predicted_value` field - which will contain the model output - as a `dense_vector` field.
85+
To review a mapping of an index for an NLP model, refer to the mapping code snippet in the [Add the text embedding model to an ingest inference pipeline](/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md#ex-text-emb-ingest) section of the related tutorial. The example shows how to create an index mapping that defines the `my_embeddings.predicted_value` field - which will contain the model output - as a `dense_vector` field.
8986

9087
```console
9188
PUT my-index
@@ -107,15 +104,15 @@ PUT my-index
107104
1. The name of the field that will contain the embeddings generated by the model.
108105
2. The field that contains the embeddings must be a `dense_vector` field.
109106
3. The model produces embeddings with a certain number of dimensions. The `dense_vector` field must be configured with the same number of dimensions by the `dims` option. Refer to the respective model documentation to get information about the number of dimensions of the embeddings.
110-
4. The name of the field from which to create the dense vector representation. In this example, the name of the field is `my_text_field`.
107+
4. The source field used to create the dense vector representation. In this example, the name of the field is `my_text_field`.
111108
5. The field type is `text` in this example.
112109
::::::
113110

114111
:::::::
115112

116113
## Generate text embeddings [deployed-generate-embeddings]
117114

118-
Once you have created the mappings for the index, you can generate text embeddings from your input text. This can be done by using an [ingest pipeline](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md) with an [inference processor](elasticsearch://reference/enrich-processor/inference-processor.md). The ingest pipeline processes the input data and indexes it into the destination index. At index time, the inference ingest processor uses the trained model to infer against the data ingested through the pipeline. After you created the ingest pipeline with the inference processor, you can ingest your data through it to generate the model output.
115+
Use an [ingest pipeline](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md) with an [inference processor](elasticsearch://reference/enrich-processor/inference-processor.md) to generate embeddings at ingest time. The ingest pipeline processes the input data and indexes it into the destination index. At index time, the inference ingest processor uses the trained model to infer against the data ingested through the pipeline. After you created the ingest pipeline with the inference processor, you can ingest your data through it to generate the model output.
119116

120117
:::::::{tab-set}
121118

@@ -144,12 +141,11 @@ PUT _ingest/pipeline/my-text-embeddings-pipeline
144141

145142
1. Configuration object that defines the `input_field` for the {{infer}} process and the `output_field` that will contain the {{infer}} results.
146143

147-
148144
To ingest data through the pipeline to generate tokens with ELSER, refer to the [Ingest the data through the {{infer}} ingest pipeline](/solutions/search/semantic-search/semantic-search-elser-ingest-pipelines.md#reindexing-data-elser) section of the tutorial. After you successfully ingested documents by using the pipeline, your index will contain the tokens generated by ELSER. Tokens are learned associations capturing relevance, they are not synonyms. To learn more about what tokens are, refer to [this page](/explore-analyze/machine-learning/nlp/ml-nlp-elser.md#elser-tokens).
149145
::::::
150146

151147
::::::{tab-item} Dense vector models
152-
This is how an ingest pipeline that uses a text embedding model is created:
148+
Example ingest pipeline using a text embedding model:
153149

154150
```console
155151
PUT _ingest/pipeline/my-text-embeddings-pipeline
@@ -172,15 +168,13 @@ PUT _ingest/pipeline/my-text-embeddings-pipeline
172168
1. The model ID of the text embedding model you want to use.
173169
2. The `field_map` object maps the input document field name (which is `my_text_field` in this example) to the name of the field that the model expects (which is always `text_field`).
174170

175-
176171
To ingest data through the pipeline to generate text embeddings with your chosen model, refer to the [Add the text embedding model to an inference ingest pipeline](/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md#ex-text-emb-ingest) section. The example shows how to create the pipeline with the inference processor and reindex your data through the pipeline. After you successfully ingested documents by using the pipeline, your index will contain the text embeddings generated by the model.
177172
::::::
178173

179174
:::::::
180175
Now it is time to perform semantic search!
181176

182-
183-
## Search the data [deployed-search]
177+
## Search the data with vector search [deployed-search]
184178

185179
Depending on the type of model you have deployed, you can query rank features with a [sparse vector](elasticsearch://reference/query-languages/query-dsl/query-dsl-sparse-vector-query.md) query, or dense vectors with a kNN search.
186180

@@ -201,6 +195,7 @@ GET my-index/_search
201195
}
202196
}
203197
```
198+
204199
::::::
205200

206201
::::::{tab-item} Dense vector models
@@ -222,6 +217,7 @@ GET my-index/_search
222217
}
223218
}
224219
```
220+
225221
::::::
226222

227223
:::::::
@@ -268,6 +264,7 @@ GET my-index/_search
268264
}
269265
}
270266
```
267+
271268
::::::
272269

273270
::::::{tab-item} Dense vector models
@@ -310,6 +307,7 @@ GET my-index/_search
310307
}
311308
}
312309
```
310+
313311
::::::
314312

315313
:::::::

solutions/search/vector/sparse-vector.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,29 +3,29 @@ applies_to:
33
stack:
44
serverless:
55
---
6-
# Sparse vector [sparse-vector-foo]
6+
# Sparse vector search in {{es}} [sparse-vector-search]
77

8-
When working with sparse vectors in {{es}}, you'll be using the Elastic learned sparse encoder (ELSER) model at index and query time to expand content with semantically related terms.
8+
When working with sparse vectors in {{es}}, you'll use the [**Elastic Learned Sparse Encoder (ELSER)**](/explore-analyze/machine-learning/nlp/ml-nlp-elser.md) at index and query time to expand content into semantically related, weighted terms.
99

1010
This approach preserves explainability while adding semantic understanding, with each document or query expanded into a set of weighted terms.
1111

1212
Sparse vector search with ELSER is ideal for:
1313

14-
- Enhanced keyword search
15-
- Cases requiring explainable results
14+
- Enhanced keyword search with semantic expansion
15+
- Use cases requiring explainable results
1616
- Domain-specific search
1717
- Large-scale deployments
1818

19-
## Working with sparse vectors in Elasticsearch
19+
## Working with sparse vectors in {{es}}
2020

2121
:::{tip}
2222
Using the `semantic_text` field type provides automatic model management and sensible defaults. [Learn more](../semantic-search/semantic-search-semantic-text.md).
2323
:::
2424

2525
Sparse vector search with ELSER expands both documents and queries into weighted terms. To use sparse vectors in {{es}}:
2626

27-
1. Index documents with ELSER
27+
1. **Index documents with ELSER**
2828
- Deploy and configure the ELSER model
2929
- Use the `sparse_vector` field type
3030
- See [this overview](../semantic-search.md#using-nlp-models) for implementation options
31-
2. Query the index using [`sparse_vector` query](elasticsearch://reference/query-languages/query-dsl/query-dsl-sparse-vector-query.md).
31+
2. **Query the index** using the [`sparse_vector` query](elasticsearch://reference/query-languages/query-dsl/query-dsl-sparse-vector-query.md).

0 commit comments

Comments
 (0)