Skip to content

Commit ec136e1

Browse files
leemthompoleemthompo
authored
[DOCS] Update esql-for-search for 8.19
8.19 equivalent of elastic/docs-content#1797
1 parent bfb1f5b commit ec136e1

File tree

1 file changed

+72
-131
lines changed

1 file changed

+72
-131
lines changed

‎docs/reference/esql/esql-for-search.asciidoc‎

Lines changed: 72 additions & 131 deletions
Original file line numberDiff line numberDiff line change
@@ -1,212 +1,153 @@
11
[[esql-for-search]]
22
=== Using {esql} for search
33

4-
preview::[]
5-
64
This page provides an overview of how to use {esql} for search use cases.
75

86
[TIP]
97
====
10-
Prefer to get started with a hands-on tutorial? Check out <<esql-search-tutorial>>.
8+
For a hands-on tutorial check out <<esql-search-tutorial>>.
119
====
1210

13-
The following table summarizes the key search features available in {esql} and when they were introduced.
11+
[[esql-search-quick-reference]]
12+
==== {esql} search quick reference
13+
14+
The following table summarizes the key search features available in {esql} and when they were introduced, organized chronologically by release.
1415

15-
[cols="1,1,2", options="header"]
16+
[cols="1,2,1", options="header"]
1617
|===
17-
|Feature |Available since |Description
18+
|Feature |Description |Available since
1819

19-
|<<esql-search-functions,Full text search functions>>
20+
|<<esql-for-search-match-function-operator,Match function/operator>>
21+
|Perform basic text searches with `MATCH` function or match operator (`:`)
2022
|8.17
21-
|Perform basic text searches with <<esql-match, match function>> and <<esql-search-operators,match operator (`:`)>>
2223

2324
|<<esql-for-search-query-string,Query string function>>
25+
|Execute complex queries with `QSTR` using Query String syntax
2426
|8.17
25-
|Execute complex queries with <<esql-qstr,`qstr`>> using Query String syntax
2627

2728
|<<esql-for-search-scoring,Relevance scoring>>
28-
|8.18/9.0
2929
|Calculate and sort by relevance with `METADATA _score`
30-
31-
|Enhanced match options
3230
|8.18/9.0
33-
|Configure text searches with additional parameters for the `match` function
34-
35-
|<<esql-for-search-kql,Kibana Query Language>>
36-
|8.18/9.0
37-
|Use Kibana Query Language with <<esql-kql,`kql`>> function
3831

3932
|<<esql-for-search-semantic,Semantic search>>
40-
|8.18/9.0
4133
|Perform semantic searches on `semantic_text` field types
34+
|8.18/9.0
4235

4336
|<<esql-for-search-hybrid,Hybrid search>>
44-
|8.18/9.0
4537
|Combine lexical and semantic search approaches with custom weights
46-
|===
38+
|8.18/9.0
4739

48-
[[esql-filtering-vs-searching]]
49-
==== Filtering vs. searching
40+
|<<esql-for-search-kql,Kibana Query Language>>
41+
|Use Kibana Query Language with the `KQL` function
42+
|8.18/9.0
5043

51-
{esql} can be used for both simple filtering and relevance-based searching:
44+
|<<esql-match-phrase-function,Match phrase function>>
45+
|Perform phrase matching with `MATCH_PHRASE` function
46+
|8.19/9.1
47+
|===
5248

53-
* **Filtering** removes non-matching documents without calculating relevance scores
54-
* **Searching** both filters documents and ranks them by how well they match the query
49+
[[how-search-works-in-esql]]
50+
==== How search works in {esql}
5551

56-
Note that filtering is faster than searching, because it doesn't require score calculations.
52+
{esql} provides two distinct approaches for finding documents: filtering and searching. Understanding the difference is crucial for building effective queries and choosing the right approach for your use case.
5753

58-
[[esql-for-search-scoring]]
59-
===== Relevance scoring
54+
**Filtering** removes documents that don't meet your criteria. It's a binary yes/no decision - documents either match your conditions or they don't. Filtering is faster because it doesn't calculate relevance scores and leverages efficient index structures for exact matches, ranges, and boolean logic.
6055

61-
To get the most relevant results first, you need to use `METADATA _score` and sort by score. For example:
56+
**Searching** both filters documents and ranks them by relevance. It calculates a score for each matching document based on how well the content matches your query, allowing you to sort results from most relevant to least relevant. Search functions use advanced text analysis including stemming, synonyms, and fuzzy matching.
6257

63-
[source,esql]
64-
----
65-
FROM books METADATA _score
66-
| WHERE match(title, "Shakespeare") OR match(plot, "Shakespeare")
67-
| SORT _score DESC
68-
----
58+
**When to choose filtering:**
59+
- Exact category matches (`category.keyword == "Electronics"`)
60+
- Date ranges (`date >= "2023-01-01"`)
61+
- Numerical comparisons (`price < 100`)
62+
- Any scenario where you want all matching results without ranking
6963

70-
[[esql-for-search-how-scoring-works]]
71-
===== How `_score` works
64+
**When to choose searching:**
65+
- Text queries where some results are more relevant than others
66+
- Finding documents similar to a search phrase
67+
- Any scenario where you want the "best" matches first
68+
- You want to use <<analysis,analyzers>> or <<search-with-synonyms,synonyms>>
7269

73-
When working with relevance scoring in ES|QL:
70+
{esql}'s search functions address several key limitations that existed for text filtering: they work directly on multivalued fields, leverage analyzers for proper text analysis, and use optimized Lucene index structures for better performance.
7471

75-
* If you don't include `METADATA _score` in your query, this only performs filtering operations with no relevance calculation.
76-
* When you include `METADATA _score`, any search function included in `WHERE` conditions contribute to the relevance score. This means that every occurrence of `MATCH`, `QSTR` and `KQL` will affect the score.
77-
* Filtering operations that are not search functions, like range conditions and exact matches, don't affect the score.
78-
* Including `METADATA _score` doesn't automatically sort your results by relevance. You must explicitly use `SORT _score DESC` or `SORT _score ASC` to order your results by relevance.
72+
[[esql-for-search-scoring]]
73+
===== Relevance scoring
7974

80-
[[esql-for-search-full-text]]
81-
==== Full text search
75+
To get relevance-ranked results, you must explicitly request scoring with `METADATA _score` and sort by the score. Without this, even search functions like `MATCH` will only filter documents without ranking them.
8276

83-
[[esql-for-search-match-function-operator]]
84-
===== Match function and operator
77+
**Without `METADATA _score`**: All operations are filtering-only, even `MATCH`, `QSTR`, and `KQL` functions. Documents either match or don't match - no ranking occurs.
8578

86-
ES|QL offers two syntax options for `match`, which replicate the functionality of <<query-dsl-match-query, `match`>> queries in Query DSL.
79+
**With `METADATA _score`**: <<esql-search-functions,Search functions>> contribute to relevance scores, while filtering operations (range conditions, exact matches) don't affect scoring. You must explicitly use `SORT _score DESC` to see the most relevant results first.
8780

88-
Use the compact operator syntax (`:`) for simple text matching with default parameters.
81+
This gives you full control over when to use fast filtering versus slower but more powerful relevance-based searching.
8982

90-
[source,esql]
91-
----
92-
FROM logs | WHERE match(message, "connection error")
93-
----
83+
[[search-functions]]
84+
==== Search functions
9485

95-
Use the `match()` function syntax when you need to pass additional parameters:
86+
The following functions provide text-based search capabilities in {esql} with different levels of precision and control.
9687

97-
[source,esql]
98-
----
99-
FROM products | WHERE match(name, "laptop", { "boost": 2.0 })
100-
----
88+
[[esql-for-search-match-function-operator]]
89+
===== `MATCH` function and operator
10190

102-
These full-text functions address several key limitations that existed for text filtering in {esql}:
91+
{esql} offers two syntax options for match, which replicate the functionality of <<query-dsl-match-query, `match`>> queries in Query DSL.
10392

104-
* They work directly on multivalued fields, returning results when any value in a multivalued field matches the query
105-
* They leverage analyzers, ensuring the query is analyzed with the same process as the indexed data (enabling case-insensitive matching, ASCII folding, stopword removal, and synonym support)
106-
* They are highly performant, using Lucene index structures rather than pattern matching or regular expressions to locate terms in your data
93+
- Use the compact <<esql-match-operator,operator syntax (:)>> for simple text matching with default parameters.
94+
- Use the <<esql-match,MATCH function syntax>> for more control over the query, such as specifying analyzers, fuzziness, and other parameters.
10795

108-
Refer to this blog for more context: https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL].
96+
Refer to the <<esql-search-tutorial,tutorial>> for examples of both syntaxes.
10997

110-
[TIP]
111-
====
112-
See <<match-field-params,Match field parameters>> for more advanced options using match.
113-
====
98+
[[esql-match-phrase-function]]
99+
===== `MATCH_PHRASE` function
114100

115-
[IMPORTANT]
116-
====
117-
These queries match documents but don't automatically sort by relevance. To get the most relevant results first, you need to use `METADATA _score` and sort by score. See <<esql-for-search-scoring,Relevance scoring>> for more information.
118-
====
101+
Use the <<esql-match_phrase,`MATCH_PHRASE` function>> to perform a `match_phrase` query on the specified field. This is equivalent to using the <<query-dsl-match-query-phrase,match_phrase query>> in Query DSL.
119102

120-
[[esql-for-search-query-string]]
121-
===== Query string function (`QSTR`)
103+
For exact phrase matching rather than individual word matching, use `MATCH_PHRASE`.
122104

123-
The <<esql-qstr,`qstr` function>> provides the same functionality as the Query DSL's `query_string` query. This is for advanced use cases, such as wildcard searches, searches across multiple fields, and more.
105+
[[esql-for-search-query-string]]
106+
===== Query string (`QSTR`) function
124107

125-
[source,esql]
126-
----
127-
FROM articles METADATA _score
128-
| WHERE QSTR("(new york city) OR (big apple)")
129-
| SORT _score DESC
130-
| LIMIT 10
131-
----
108+
The <<esql-qstr,`qstr` function>> provides the same functionality as the Query DSL's `query_string` query. This enables advanced search patterns with wildcards, boolean logic, and multi-field searches.
132109

133110
For complete details, refer to the <<query-dsl-query-string-query, Query DSL `query_string` docs>>.
134111

135112
[[esql-for-search-kql]]
136-
===== Kibana Query Language function (`KQL`)
113+
===== `KQL` function
137114

138-
Use the <<esql-kql,KQL function>> to use the {kibana-ref}/kuery-query.html[Kibana Query Language] in your ES|QL queries:
115+
Use the <<esql-kql,KQL function>> to use the {kibana-ref}/kuery-query.html[Kibana Query Language] in your {esql} queries.
139116

140-
[source,esql]
141-
----
142-
FROM logs*
143-
| WHERE KQL("http.request.method:GET AND agent.type:filebeat")
144-
----
117+
For migrating queries from other Kibana interfaces, the `KQL` function preserves existing query syntax and allows gradual migration to {esql} without rewriting existing Kibana queries.
145118

146-
The `kql` function is useful when transitioning queries from Kibana's Discover, Dashboard, or other interfaces that use KQL. This will allow you to gradually migrate queries to ES|QL without needing to rewrite them all at once.
119+
[[advanced-search-capabilities]]
120+
==== Advanced search capabilities
147121

148122
[[esql-for-search-semantic]]
149-
==== Semantic search
150-
151-
You can perform semantic searches over <<semantic-text, `semantic_text`>> field types using the same match syntax as full-text search.
152-
153-
This example uses the match operator `:`:
123+
===== Semantic search
154124

155-
[source,esql]
156-
----
157-
FROM articles METADATA _score
158-
| WHERE semantic_content:"What are the impacts of climate change on agriculture?"
159-
| SORT _score DESC
160-
----
125+
<<semantic-search,Semantic search>> leverages machine learning models to understand the meaning of text, enabling more accurate and context-aware search results.
161126

162-
This example uses the match function:
127+
In {esql}, you can perform semantic searches on <<semantic-text, `semantic_text`>> field types using the same match syntax as full-text search.
163128

164-
[source,esql]
165-
----
166-
FROM articles METADATA _score
167-
| WHERE match(semantic_content, "What are the impacts of climate change on agriculture?")
168-
| SORT _score DESC
169-
----
129+
Refer to <<semantic-search-semantic-text,semantic search with semantic_text>> for an example or follow the <<esql-search-tutorial,tutorial>>.
170130

171131
[[esql-for-search-hybrid]]
172-
==== Hybrid search
132+
===== Hybrid search
173133

174-
Combine traditional and semantic search with custom weights:
134+
<<hybrid-search,Hybrid search>> combines lexical and semantic search with custom weights to leverage both exact keyword matching and semantic understanding.
175135

176-
[source,esql]
177-
----
178-
FROM books METADATA _score
179-
| WHERE match(semantic_title, "fantasy adventure", { "boost": 0.75 })
180-
OR match(title, "fantasy adventure", { "boost": 0.25 })
181-
| SORT _score DESC
182-
----
183-
184-
[[esql-for-search-limitations]]
185-
==== Limitations
186-
187-
Refer to <<esql-limitations-full-text-search, {esql} limitations>> for a list of known limitations.
136+
Refer to <<hybrid-semantic-text,hybrid search with semantic_text>> for an example or follow the <<esql-search-tutorial,tutorial>>.
188137

189138
[[esql-for-search-next-steps]]
190139
==== Next steps
191140

192141
[[esql-for-search-tutorials]]
193142
===== Tutorials and how-to guides
194143

195-
* <<esql-search-tutorial>>: Hands-on tutorial for getting started with search tools in {esql}
196-
* <<semantic-search-semantic-text>>: Learn how to use the `semantic_text` field type
144+
- <<esql-search-tutorial>>: Hands-on tutorial for getting started with search tools in {esql}, with concrete examples of the functionalities described in this page
197145

198146
[[esql-for-search-reference]]
199147
===== Technical reference
200148

201-
* <<esql-search-functions>>: Complete reference for all search functions
202-
* <<esql-limitations-full-text-search, Limitations>>: Current limitations for search in ES|QL
203-
204-
[[esql-for-search-concepts]]
205-
===== Background concepts
206-
207-
* <<analysis>>: Learn how text is processed for full-text search
208-
* <<semantic-search>>: Get an overview of semantic search in {es}
209-
* <<query-filter-context>>: Understand the difference between query and filter contexts in {es}
149+
- <<esql-search-functions>>: Complete reference for all search functions
150+
- <<esql-limitations-full-text-search, Limitations>>: Current limitations for search functions in {esql}
210151

211152
[[esql-for-search-blogs]]
212153
===== Related blog posts

0 commit comments

Comments
 (0)