Skip to content

Commit 69115b5

Browse files
leemthompoleemthompo
authored
[DOCS][8.x] Add Using ESQL for search landing page (elastic#125580) (elastic#125653)
1 parent bb43a7d commit 69115b5

File tree

5 files changed

+266
-32
lines changed

5 files changed

+266
-32
lines changed

‎docs/reference/esql/esql-for-search.asciidoc‎

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
[[esql-for-search]]
2+
=== Using {esql} for search
3+
4+
preview::[]
5+
6+
This page provides an overview of how to use {esql} for search use cases.
7+
8+
// [TIP]
9+
// ====
10+
// Prefer to get started with a hands-on tutorial? Check out <<esql-search-tutorial>>.
11+
// ====
12+
13+
The following table summarizes the key search features available in {esql} and when they were introduced.
14+
15+
[cols="1,1,2", options="header"]
16+
|===
17+
|Feature |Available since |Description
18+
19+
|<<esql-search-functions,Full text search functions>>
20+
|8.17
21+
|Perform basic text searches with <<esql-match, match function>> and <<esql-search-operators,match operator (`:`)>>
22+
23+
|<<esql-for-search-query-string,Query string function>>
24+
|8.17
25+
|Execute complex queries with <<esql-qstr,`qstr`>> using Query String syntax
26+
27+
|<<esql-for-search-scoring,Relevance scoring>>
28+
|8.18/9.0
29+
|Calculate and sort by relevance with `METADATA _score`
30+
31+
|Enhanced match options
32+
|8.18/9.0
33+
|Configure text searches with additional parameters for the `match` function
34+
35+
|<<esql-for-search-kql,Kibana Query Language>>
36+
|8.18/9.0
37+
|Use Kibana Query Language with <<esql-kql,`kql`>> function
38+
39+
|<<esql-for-search-semantic,Semantic search>>
40+
|8.18/9.0
41+
|Perform semantic searches on `semantic_text` field types
42+
43+
|<<esql-for-search-hybrid,Hybrid search>>
44+
|8.18/9.0
45+
|Combine lexical and semantic search approaches with custom weights
46+
|===
47+
48+
[[esql-filtering-vs-searching]]
49+
==== Filtering vs. searching
50+
51+
{esql} can be used for both simple filtering and relevance-based searching:
52+
53+
* **Filtering** removes non-matching documents without calculating relevance scores
54+
* **Searching** both filters documents and ranks them by how well they match the query
55+
56+
Note that filtering is faster than searching, because it doesn't require score calculations.
57+
58+
[[esql-for-search-scoring]]
59+
===== Relevance scoring
60+
61+
To get the most relevant results first, you need to use `METADATA _score` and sort by score. For example:
62+
63+
[source,esql]
64+
----
65+
FROM books METADATA _score
66+
| WHERE match(title, "Shakespeare") OR match(plot, "Shakespeare")
67+
| SORT _score DESC
68+
----
69+
70+
[[esql-for-search-how-scoring-works]]
71+
===== How `_score` works
72+
73+
When working with relevance scoring in ES|QL:
74+
75+
* If you don't include `METADATA _score` in your query, this only performs filtering operations with no relevance calculation.
76+
* When you include `METADATA _score`, any search function included in `WHERE` conditions contribute to the relevance score. This means that every occurrence of `MATCH`, `QSTR` and `KQL` will affect the score.
77+
* Filtering operations that are not search functions, like range conditions and exact matches, don't affect the score.
78+
* Including `METADATA _score` doesn't automatically sort your results by relevance. You must explicitly use `SORT _score DESC` or `SORT _score ASC` to order your results by relevance.
79+
80+
[[esql-for-search-full-text]]
81+
==== Full text search
82+
83+
[[esql-for-search-match-function-operator]]
84+
===== Match function and operator
85+
86+
ES|QL offers two syntax options for `match`, which replicate the functionality of <<query-dsl-match-query, `match`>> queries in Query DSL.
87+
88+
Use the compact operator syntax (`:`) for simple text matching with default parameters.
89+
90+
[source,esql]
91+
----
92+
FROM logs | WHERE match(message, "connection error")
93+
----
94+
95+
Use the `match()` function syntax when you need to pass additional parameters:
96+
97+
[source,esql]
98+
----
99+
FROM products | WHERE match(name, "laptop", { "boost": 2.0 })
100+
----
101+
102+
These full-text functions address several key limitations that existed for text filtering in {esql}:
103+
104+
* They work directly on multivalued fields, returning results when any value in a multivalued field matches the query
105+
* They leverage analyzers, ensuring the query is analyzed with the same process as the indexed data (enabling case-insensitive matching, ASCII folding, stopword removal, and synonym support)
106+
* They are highly performant, using Lucene index structures rather than pattern matching or regular expressions to locate terms in your data
107+
108+
Refer to this blog for more context: https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL].
109+
110+
[TIP]
111+
====
112+
See <<match-field-params,Match field parameters>> for more advanced options using match.
113+
====
114+
115+
[IMPORTANT]
116+
====
117+
These queries match documents but don't automatically sort by relevance. To get the most relevant results first, you need to use `METADATA _score` and sort by score. See <<esql-for-search-scoring,Relevance scoring>> for more information.
118+
====
119+
120+
[[esql-for-search-query-string]]
121+
===== Query string function (`QSTR`)
122+
123+
The <<esql-qstr,`qstr` function>> provides the same functionality as the Query DSL's `query_string` query. This is for advanced use cases, such as wildcard searches, searches across multiple fields, and more.
124+
125+
[source,esql]
126+
----
127+
FROM articles METADATA _score
128+
| WHERE QSTR("(new york city) OR (big apple)")
129+
| SORT _score DESC
130+
| LIMIT 10
131+
----
132+
133+
For complete details, refer to the <<query-dsl-query-string-query, Query DSL `query_string` docs>>.
134+
135+
[[esql-for-search-kql]]
136+
===== Kibana Query Language function (`KQL`)
137+
138+
Use the <<esql-kql,KQL function>> to use the {kibana-ref}/kuery-query.html[Kibana Query Language] in your ES|QL queries:
139+
140+
[source,esql]
141+
----
142+
FROM logs*
143+
| WHERE KQL("http.request.method:GET AND agent.type:filebeat")
144+
----
145+
146+
The `kql` function is useful when transitioning queries from Kibana's Discover, Dashboard, or other interfaces that use KQL. This will allow you to gradually migrate queries to ES|QL without needing to rewrite them all at once.
147+
148+
[[esql-for-search-semantic]]
149+
==== Semantic search
150+
151+
You can perform semantic searches over <<semantic-text, `semantic_text`>> field types using the same match syntax as full-text search.
152+
153+
This example uses the match operator `:`:
154+
155+
[source,esql]
156+
----
157+
FROM articles METADATA _score
158+
| WHERE semantic_content:"What are the impacts of climate change on agriculture?"
159+
| SORT _score DESC
160+
----
161+
162+
This example uses the match function:
163+
164+
[source,esql]
165+
----
166+
FROM articles METADATA _score
167+
| WHERE match(semantic_content, "What are the impacts of climate change on agriculture?")
168+
| SORT _score DESC
169+
----
170+
171+
[[esql-for-search-hybrid]]
172+
==== Hybrid search
173+
174+
Combine traditional and semantic search with custom weights:
175+
176+
[source,esql]
177+
----
178+
FROM books METADATA _score
179+
| WHERE match(semantic_title, "fantasy adventure", { "boost": 0.75 })
180+
OR match(title, "fantasy adventure", { "boost": 0.25 })
181+
| SORT _score DESC
182+
----
183+
184+
[[esql-for-search-limitations]]
185+
==== Limitations
186+
187+
Refer to <<esql-limitations-full-text-search, {esql} limitations>> for a list of known limitations.
188+
189+
[[esql-for-search-next-steps]]
190+
==== Next steps
191+
192+
[[esql-for-search-tutorials]]
193+
===== Tutorials and how-to guides
194+
195+
// TODO * <<esql-search-tutorial>>: Hands-on tutorial for getting started with search tools in {esql}
196+
* <<semantic-search-semantic-text>>: Learn how to use the `semantic_text` field type
197+
198+
[[esql-for-search-reference]]
199+
===== Technical reference
200+
201+
* <<esql-search-functions>>: Complete reference for all search functions
202+
* <<esql-limitations-full-text-search, Limitations>>: Current limitations for search in ES|QL
203+
204+
[[esql-for-search-concepts]]
205+
===== Background concepts
206+
207+
* <<analysis>>: Learn how text is processed for full-text search
208+
* <<semantic-search>>: Get an overview of semantic search in {es}
209+
* <<query-filter-context>>: Understand the difference between query and filter contexts in {es}
210+
211+
[[esql-for-search-blogs]]
212+
===== Related blog posts
213+
214+
// TODO* https://www.elastic.co/blog/esql-you-know-for-search-scoring-semantic-search[ES|QL, you know for Search]: Introducing scoring and semantic search
215+
* https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL]: Overview of text filtering capabilities

‎docs/reference/esql/esql-get-started.asciidoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
<titleabbrev>Getting started</titleabbrev>
55
++++
66

7-
This guide shows how you can use {esql} to query and aggregate your data.
7+
This guide shows how you can use {esql} to query and aggregate your data. Refer to <<esql-for-search>> if you'd like to learn more about using {esql} for search use cases.
88

99
[TIP]
1010
====

‎docs/reference/esql/esql-language.asciidoc‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ Detailed reference documentation for the {esql} language:
1111
* <<esql-functions>>
1212
* <<esql-metadata-fields>>
1313
* <<esql-multivalued-fields>>
14-
* <<esql-enrich-data>>
1514
* <<esql-lookup-join>>
15+
* <<esql-enrich-data>>
1616
* <<esql-process-data-with-dissect-and-grok>>
1717
* <<esql-implicit-casting>>
1818
* <<esql-time-spans>>
@@ -23,7 +23,7 @@ include::esql-functions-operators.asciidoc[]
2323
include::metadata-fields.asciidoc[]
2424
include::multivalued-fields.asciidoc[]
2525
include::esql-process-data-with-dissect-grok.asciidoc[]
26-
include::esql-enrich-data.asciidoc[]
2726
include::esql-lookup-join.asciidoc[]
27+
include::esql-enrich-data.asciidoc[]
2828
include::implicit-casting.asciidoc[]
2929
include::time-spans.asciidoc[]

‎docs/reference/esql/esql-using.asciidoc‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,33 @@
11
[[esql-using]]
22
== Using {esql}
33

4+
This page is an overview of the various ways you can use {esql} across different Elastic interfaces and use cases.
5+
46
<<esql-rest>>::
5-
Information about using the <<esql-apis,{esql} query APIs>>.
7+
Learn how to use the <<esql-apis,{esql} query APIs>>.
8+
9+
<<esql-for-search>>::
10+
Learn how to use {esql} for search use cases.
611

712
<<esql-kibana>>::
8-
Using {esql} in {kib} to query and aggregate your data, create visualizations,
13+
Learn how to use {esql} in {kib} to query and aggregate your data, create visualizations,
914
and set up alerts.
1015

1116
<<esql-elastic-security>>::
12-
Using {esql} in {elastic-sec} to investigate events in Timeline, create
17+
Learn how to use {esql} in {elastic-sec} to investigate events in Timeline, create
1318
detection rules, and build {esql} queries using Elastic AI Assistant.
1419

1520
<<esql-multi-index>>::
16-
Using {esql} to query multiple indexes and resolve field type mismatches.
21+
Learn how to use {esql} to query multiple indexes and resolve field type mismatches.
1722

1823
<<esql-cross-clusters>>::
19-
Using {esql} to query across multiple clusters.
24+
Learn how to use {esql} to query across multiple clusters.
2025

2126
<<esql-task-management>>::
22-
Using the <<tasks,task management API>> to list and cancel {esql} queries.
27+
Learn how to use the <<tasks,task management API>> to list and cancel {esql} queries.
2328

2429
include::esql-rest.asciidoc[]
30+
include::esql-for-search.asciidoc[]
2531
include::esql-kibana.asciidoc[]
2632
include::esql-security-solution.asciidoc[]
2733
include::esql-multi-index.asciidoc[]

‎docs/reference/esql/index.asciidoc‎

Lines changed: 36 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -22,21 +22,7 @@ a series of operations, where the output of one operation becomes the input for
2222
the next, enabling complex data transformations and analysis.
2323

2424
[discrete]
25-
=== The {esql} Compute Engine
26-
27-
{esql} is more than a language: it represents a significant investment in new
28-
compute capabilities within {es}. To achieve both the functional and performance
29-
requirements for {esql}, it was necessary to build an entirely new compute
30-
architecture. {esql} search, aggregation, and transformation functions are
31-
directly executed within Elasticsearch itself. Query expressions are not
32-
transpiled to Query DSL for execution. This approach allows {esql} to be
33-
extremely performant and versatile.
34-
35-
The new {esql} execution engine was designed with performance in mind — it
36-
operates on blocks at a time instead of per row, targets vectorization and cache
37-
locality, and embraces specialization and multi-threading. It is a separate
38-
component from the existing Elasticsearch aggregation framework with different
39-
performance characteristics.
25+
=== Documentation organization
4026

4127
The {esql} documentation is organized in these sections:
4228

@@ -45,23 +31,33 @@ A tutorial to help you get started with {esql}.
4531

4632
<<esql-language>>::
4733

48-
Reference documentation for the <<esql-syntax,{esql} syntax>>,
49-
<<esql-commands,commands>>, and <<esql-functions-operators,functions and
50-
operators>>. Information about working with <<esql-metadata-fields,metadata
51-
fields>> and <<esql-multivalued-fields,multivalued fields>>. And guidance for
52-
<<esql-process-data-with-dissect-and-grok,data processing with DISSECT and
53-
GROK>> and <<esql-enrich-data,data enrichment with ENRICH>>.
34+
Reference documentation for the <<esql-syntax,{esql} syntax>>:
35+
36+
* Reference for <<esql-commands,commands>>, and <<esql-functions-operators,functions and
37+
operators>>
38+
* How to work with <<esql-metadata-fields,metadata
39+
fields>> and <<esql-multivalued-fields,multivalued fields>>
40+
* How to work with
41+
<<esql-process-data-with-dissect-and-grok,DISSECT and
42+
GROK>>, <<esql-enrich-data,ENRICH>>, and <<esql-lookup-join,LOOKUP join>>
5443

5544
<<esql-using>>::
56-
An overview of using the <<esql-rest>>, <<esql-kibana>>,
57-
<<esql-elastic-security>>, <<esql-cross-clusters>>, and <<esql-task-management>>.
45+
An overview of:
46+
* <<esql-rest,Using the {esql} rest API>>
47+
* <<esql-for-search>>
48+
* <<esql-kibana>>
49+
* <<esql-elastic-security>>
50+
* <<esql-cross-clusters>>
51+
* <<esql-task-management>>
5852

5953
<<esql-limitations>>::
6054
The current limitations of {esql}.
6155

6256
<<esql-examples>>::
6357
A few examples of what you can do with {esql}.
6458

59+
60+
6561
include::esql-get-started.asciidoc[]
6662

6763
include::esql-language.asciidoc[]
@@ -74,3 +70,20 @@ include::esql-examples.asciidoc[]
7470

7571
:esql-tests!:
7672
:esql-specs!:
73+
74+
[discrete]
75+
=== The {esql} Compute Engine
76+
77+
{esql} is more than a language: it represents a significant investment in new
78+
compute capabilities within {es}. To achieve both the functional and performance
79+
requirements for {esql}, it was necessary to build an entirely new compute
80+
architecture. {esql} search, aggregation, and transformation functions are
81+
directly executed within Elasticsearch itself. Query expressions are not
82+
transpiled to Query DSL for execution. This approach allows {esql} to be
83+
extremely performant and versatile.
84+
85+
The new {esql} execution engine was designed with performance in mind — it
86+
operates on blocks at a time instead of per row, targets vectorization and cache
87+
locality, and embraces specialization and multi-threading. It is a separate
88+
component from the existing Elasticsearch aggregation framework with different
89+
performance characteristics.

0 commit comments

Comments
 (0)