You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: solutions/search/get-started/semantic-search.md
+76-31Lines changed: 76 additions & 31 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -41,17 +41,25 @@ The way that you store vectors has a significant impact on the performance and a
41
41
They must be stored in specialized data structures designed to ensure efficient similarity search and speedy vector distance calculations.
42
42
This guide uses the [semantic text field type](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md), which provides sensible defaults and automation.
43
43
44
-
Try vectorizing a small set of documents.
44
+
:::::{stepper}
45
+
::::{step} Create an index
46
+
An index is a collection of documents uniquely identified by a name or an alias.
45
47
You can follow the guided index workflow:
46
48
47
49
- If you're using {{es-serverless}}, go to **{{es}} > Home**, select the semantic search workflow, and click **Create a semantic optimized index**.
48
50
- If you're running {{es}} locally, go to **{{es}} > Home** and click **Create API index**. Select the semantic search workflow.
49
51
50
-
Alternatively, run the following API requests in the [Console](/explore-analyze/query-filter/tools/console.md):
52
+
When you complete the workflow, you will have sample data and can skip to the steps related to exploring and searching it.
53
+
Alternatively, run the following API request in the [Console](/explore-analyze/query-filter/tools/console.md):
51
54
52
-
:::::{stepper}
53
-
::::{step} Create a semantic_text field mapping
55
+
```console
56
+
PUT /semantic-index
57
+
```
54
58
59
+
For an introduction to the concept of indices, check out [](/manage-data/data-store/index-basics.md).
60
+
::::
61
+
::::{step} Create a semantic_text field mapping
62
+
Each index has mappings that define how data is stored and indexed, like a schema in a relational database.
55
63
The following example creates a mapping for a single field ("content"):
56
64
57
65
```console
@@ -87,20 +95,23 @@ POST /_bulk?pretty
87
95
88
96
The bulk ingestion might take longer than the default request timeout.
89
97
If it times out, wait for the ELSER model to load (typically 1-5 minutes) then retry it.
98
+
If you're using {{es-serverless}}, you can check the model state in **{{project-settings}} > {{models-app}}**.
99
+
<!--
100
+
TBD: Stack app location
101
+
-->
90
102
91
-
What just happened?
92
-
First, the content was divided into smaller, manageable chunks to ensure that meaningful segments can be more effectively processed and searched.
93
-
Then each chunk of text was transformed into a sparse vector by using the ELSER model's text expansion techniques.
103
+
First, the content is divided into smaller, manageable chunks to ensure that meaningful segments can be more effectively processed and searched.
104
+
Each chunk of text is then transformed into a sparse vector by using the ELSER model's text expansion techniques.
The vectors are stored in {{es}} and semantic search can now occur.
108
+
The vectors are stored in {{es}} and are ready to be used for semantic search.
98
109
::::
99
110
::::{step} Explore the data
100
111
101
112
To familiarize yourself with this data set, open [Discover](/explore-analyze/discover.md) from the navigation menu or by using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
102
113
103
-
In **Discover**, you can click the expand icon  to show details about documents in the table.
114
+
In **Discover**, you can click the expand icon  to show details about documents in the table.
@@ -113,46 +124,80 @@ For more tips, check out [](/explore-analyze/discover/discover-get-started.md).
113
124
114
125
## Test semantic search
115
126
127
+
When you run a semantic search, the text in your query must be turned into vectors that use the same embedding model as your vector database.
128
+
This step is performed automatically when you use `semantic_text` fields.
129
+
You therefore only need to pick a query language and a method for comparing the vectors.
130
+
131
+
:::::{stepper}
132
+
::::{step} Choose a query language
133
+
116
134
{{es}} provides a variety of query languages for interacting with your data.
117
135
For an overview of their features and use cases, check out [](/explore-analyze/query-filter/languages.md).
136
+
The [Elasticsearch Query Language](/explore-analyze/query-filter/languages/esql.md) (ES|QL) is designed to be easy to read and write.
137
+
It enables you to query your data directly in **Discover**, so it's a good one to start with.
118
138
119
-
You can search data that is stored in `semantic_text` fields by using a specific subset of queries, including `knn`, `match`, `semantic`, and `sparse_vector`.
120
-
The query is translated automatically into the appropriate vector representation to run against the contents of the semantic text field.
121
-
The search results include a relevance score, which measures how well each document matches the query.
122
-
123
-
Let's test a semantic search query in [Elasticsearch Query Language](/explore-analyze/query-filter/languages/esql.md) (ES|QL).
124
139
Go to **Discover** and select **Try ES|QL** from the application menu bar.
125
-
Think of some queries that are relevant to the documents you explored, such as finding the biggest park or the best for rappelling.
126
-
For example, copy the following query:
140
+
::::
141
+
::::{step} Choose a vector comparison method
142
+
You can search data that is stored in `semantic_text` fields by using a specific subset of queries, including `knn`, `match`, `semantic`, and `sparse_vector`.
143
+
For the definitive list of supported queries, refer to [](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md).
144
+
145
+
In ES|QL, you can perform semantic searches on `semantic_text` field types using the same match syntax as full-text search.
146
+
For example:
147
+
148
+
```esql
149
+
FROM semantic-index <1>
150
+
| WHERE content: "what's the biggest park?" <2>
151
+
| LIMIT 10 <3>
152
+
```
153
+
154
+
1. The FROM source command returns a table of data from the specified index.
155
+
2. A simplified syntax for the MATCH search function, this command performs a semantic query on the specified field. Think of some queries that are relevant to the documents you explored, such as finding the biggest park or the best for rappelling.
156
+
3. The LIMIT processing command defines the maximum number of rows to return.
157
+
158
+
When you click **▶Run**, the results appear in a table.
159
+
Each row in the table represents a document.
160
+
<!--
161
+
TBD: Run the same query in Console
162
+
-->
163
+
To learn more about these commands, refer to [](elasticsearch://reference/query-languages/esql/esql-syntax-reference.md) and [](/solutions/search/esql-for-search.md).
164
+
::::
165
+
::::{step} Analyze the results
166
+
167
+
To have a better understanding of how well each document matches your query, add commands to include the relevance score and sort the results based on that value.
168
+
For example:
127
169
128
170
```esql
129
171
FROM semantic-index METADATA _score <1>
130
-
| WHERE content: "what's the biggest park?" <2>
131
-
| KEEP content, _score <3>
132
-
| SORT _score DESC <4>
133
-
| LIMIT 1000 <5>
172
+
| WHERE content: "best spot for rappelling"
173
+
| KEEP content, _score <2>
174
+
| SORT _score DESC <3>
175
+
| LIMIT 10
134
176
```
135
177
136
-
1. The FROM source command returns a table of data. Each row in the table represents a document. The `METADATA` clause provides access to the query relevance score, which is a [metadata field](elasticsearch://reference/query-languages/esql/esql-metadata-fields.md).
137
-
2. A simplified syntax for the [match](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md#esql-match) search function, this command performs a semantic query on the specified field.
138
-
3. The KEEP processing command affects the columns and their order in the results table.
139
-
4. The results are sorted in descending order based on the `_score`.
140
-
5. This optional command defines the maximum number of rows to return.
178
+
1. The `METADATA` clause provides access to the query relevance score, which is a [metadata field](elasticsearch://reference/query-languages/esql/esql-metadata-fields.md).
179
+
2. The KEEP processing command affects the columns and their order in the results table.
180
+
3. The results are sorted in descending order based on the `_score`.
141
181
142
-
After you click **▶Run**, the results appear in a table.
143
-
In this example, the first row in the table is the document related to Yellowstone National Park, which had the highest relevance score for the query.
182
+
:::{tip}
183
+
Click the **ES|QL help** button to open the in-product reference documentation for all commands and functions or to get recommended queries that will help you get started.
In this example, the first row in the table is the document related to Rocky Mountain National Park, which had the highest relevance score for the query.
192
+
193
+
For more tips, check out [Using ES|QL in Discover](/explore-analyze/discover/try-esql.md).
194
+
::::
195
+
:::::
196
+
150
197
<!--
151
-
TBD: Run the same query in Console
198
+
TBD: Delete index and stop model?
152
199
-->
153
200
154
-
To learn more, check out [](/explore-analyze/discover/try-esql.md) and [](/solutions/search/esql-for-search.md).
155
-
156
201
## Next steps
157
202
158
203
Thanks for taking the time to try out semantic search in {{es-serverless}}.
0 commit comments