-
Notifications
You must be signed in to change notification settings - Fork 25.5k
[DOCS][8.x] Add Using ESQL for search
landing page
#125580
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
e8b83cb
[DOCS][8.x] Add Search for ESQL landing page
leemthompo 2097198
tweak wording
leemthompo 489e126
Revamp per Carlos' review
leemthompo 2813ba7
fix typo
leemthompo 9b108a0
make heading consistent
leemthompo b6c2207
mention filtering is faster than searching
leemthompo 1e9b31f
add space
leemthompo fa88d92
Add tech preview banner
leemthompo bdd1305
move to "Using ESQL" section with improved title
leemthompo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,212 @@ | ||
[[esql-for-search]] | ||
=== Search with {esql} | ||
|
||
This page provides an overview of how to use {esql} for search use cases. | ||
|
||
// [TIP] | ||
// ==== | ||
// Prefer to get started with a hands-on tutorial? Check out <<esql-search-tutorial>>. | ||
// ==== | ||
|
||
The following table summarizes the key search features available in {esql} and when they were introduced. | ||
|
||
[cols="1,1,2", options="header"] | ||
|=== | ||
|Feature |Available since |Description | ||
|
||
|<<esql-search-functions,Full text search functions>> | ||
|8.17 | ||
|Perform basic text searches with <<esql-match, match function>> and <<esql-search-operators,match operator (`:`)>> | ||
|
||
|<<esql-for-search-query-string,Query string function>> | ||
|8.17 | ||
|Execute complex queries with <<esql-qstr,`qstr`>> using Query String syntax | ||
|
||
|<<esql-for-search-scoring,Relevance scoring>> | ||
|8.18/9.0 | ||
|Calculate and sort by relevance with `METADATA _score` | ||
|
||
|Enhanced match options | ||
|8.18/9.0 | ||
|Configure text searches with additional parameters for the `match` function | ||
|
||
|<<esql-for-search-kql,Kibana Query Language>> | ||
|8.18/9.0 | ||
|Use Kibana Query Language with <<esql-kql,`kql`>> function | ||
|
||
|<<esql-for-search-semantic,Semantic search>> | ||
|8.18/9.0 | ||
|Perform semantic searches on `semantic_text` field types | ||
|
||
|<<esql-for-search-hybrid,Hybrid search>> | ||
|8.18/9.0 | ||
|Combine lexical and semantic search approaches with custom weights | ||
|=== | ||
|
||
[[esql-filtering-vs-searching]] | ||
==== Filtering vs. searching | ||
|
||
{esql} can be used for both simple filtering and relevance-based searching: | ||
|
||
* **Filtering** removes non-matching documents without calculating relevance scores | ||
* **Searching** both filters documents and ranks them by how well they match the query | ||
|
||
leemthompo marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Note that filtering is faster than searching, because it doesn't require score calculations. | ||
[[esql-for-search-scoring]] | ||
leemthompo marked this conversation as resolved.
Show resolved
Hide resolved
|
||
===== Relevance scoring | ||
|
||
To get the most relevant results first, you need to use `METADATA _score` and sort by score. For example: | ||
|
||
[source,esql] | ||
---- | ||
FROM books METADATA _score | ||
| WHERE match(title, "Shakespeare") OR match(plot, "Shakespeare") | ||
| SORT _score DESC | ||
---- | ||
|
||
[[esql-for-search-how-scoring-works]] | ||
===== How `_score` works | ||
|
||
When working with relevance scoring in ES|QL: | ||
|
||
* If you don't include `METADATA _score` in your query, this only performs filtering operations with no relevance calculation. | ||
* When you include `METADATA _score`, any search function included in `WHERE` conditions contribute to the relevance score. This means that every occurrence of `MATCH`, `QSTR` and `KQL` will affect the score. | ||
* Filtering operations that are not search functions, like range conditions and exact matches, don't affect the score. | ||
* Including `METADATA _score` doesn't automatically sort your results by relevance. You must explicitly use `SORT _score DESC` or `SORT _score ASC` to order your results by relevance. | ||
|
||
[[esql-for-search-full-text]] | ||
==== Full text search | ||
|
||
leemthompo marked this conversation as resolved.
Show resolved
Hide resolved
|
||
[[esql-for-search-match-function-operator]] | ||
===== Match function and operator | ||
|
||
ES|QL offers two syntax options for `match`, which replicate the functionality of <<query-dsl-match-query, `match`>> queries in Query DSL. | ||
|
||
Use the compact operator syntax (`:`) for simple text matching with default parameters. | ||
|
||
[source,esql] | ||
---- | ||
FROM logs | WHERE match(message, "connection error") | ||
---- | ||
|
||
Use the `match()` function syntax when you need to pass additional parameters: | ||
|
||
[source,esql] | ||
---- | ||
FROM products | WHERE match(name, "laptop", { "boost": 2.0 }) | ||
---- | ||
|
||
These full-text functions address several key limitations that existed for text filtering in {esql}: | ||
|
||
* They work directly on multivalued fields, returning results when any value in a multivalued field matches the query | ||
* They leverage analyzers, ensuring the query is analyzed with the same process as the indexed data (enabling case-insensitive matching, ASCII folding, stopword removal, and synonym support) | ||
* They are highly performant, using Lucene index structures rather than pattern matching or regular expressions to locate terms in your data | ||
|
||
Refer to this blog for more context: https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL]. | ||
|
||
[TIP] | ||
==== | ||
See <<match-field-params,Match field parameters>> for more advanced options using match. | ||
==== | ||
|
||
[IMPORTANT] | ||
==== | ||
These queries match documents but don't automatically sort by relevance. To get the most relevant results first, you need to use `METADATA _score` and sort by score. See <<esql-for-search-scoring,Relevance scoring>> for more information. | ||
==== | ||
|
||
[[esql-for-search-query-string]] | ||
===== Query string function (`QSTR`) | ||
|
||
The <<esql-qstr,`qstr` function>> provides the same functionality as the Query DSL's `query_string` query. This is for advanced use cases, such as wildcard searches, searches across multiple fields, and more. | ||
|
||
[source,esql] | ||
---- | ||
FROM articles METADATA _score | ||
| WHERE QSTR("(new york city) OR (big apple)") | ||
| SORT _score DESC | ||
| LIMIT 10 | ||
---- | ||
|
||
For complete details, refer to the <<query-dsl-query-string-query, Query DSL `query_string` docs>>. | ||
|
||
leemthompo marked this conversation as resolved.
Show resolved
Hide resolved
|
||
[[esql-for-search-kql]] | ||
===== Kibana Query Language function (`KQL`) | ||
|
||
Use the <<esql-kql,KQL function>> to use the {kibana-ref}/kuery-query.html[Kibana Query Language] in your ES|QL queries: | ||
|
||
[source,esql] | ||
---- | ||
FROM logs* | ||
| WHERE KQL("http.request.method:GET AND agent.type:filebeat") | ||
---- | ||
|
||
The `kql` function is useful when transitioning queries from Kibana's Discover, Dashboard, or other interfaces that use KQL. This will allow you to gradually migrate queries to ES|QL without needing to rewrite them all at once. | ||
|
||
[[esql-for-search-semantic]] | ||
==== Semantic search | ||
|
||
You can perform semantic searches over <<semantic-text, `semantic_text`>> field types using the same match syntax as full-text search. | ||
|
||
This example uses the match operator `:`: | ||
|
||
[source,esql] | ||
---- | ||
FROM articles METADATA _score | ||
| WHERE semantic_content:"What are the impacts of climate change on agriculture?" | ||
| SORT _score DESC | ||
---- | ||
|
||
This example uses the match function: | ||
|
||
[source,esql] | ||
---- | ||
FROM articles METADATA _score | ||
| WHERE match(semantic_content, "What are the impacts of climate change on agriculture?") | ||
| SORT _score DESC | ||
---- | ||
|
||
[[esql-for-search-hybrid]] | ||
==== Hybrid search | ||
|
||
Combine traditional and semantic search with custom weights: | ||
|
||
[source,esql] | ||
---- | ||
FROM books METADATA _score | ||
| WHERE match(semantic_title, "fantasy adventure", { "boost": 0.75 }) | ||
OR match(title, "fantasy adventure", { "boost": 0.25 }) | ||
| SORT _score DESC | ||
---- | ||
|
||
[[esql-for-search-limitations]] | ||
==== Limitations | ||
|
||
Refer to <<esql-limitations-full-text-search, {esql} limitations>> for a list of known limitations. | ||
|
||
[[esql-for-search-next-steps]] | ||
==== Next steps | ||
|
||
[[esql-for-search-tutorials]] | ||
===== Tutorials and how-to guides | ||
|
||
// TODO * <<esql-search-tutorial>>: Hands-on tutorial for getting started with search tools in {esql} | ||
* <<semantic-search-semantic-text>>: Learn how to use the `semantic_text` field type | ||
|
||
[[esql-for-search-reference]] | ||
===== Technical reference | ||
|
||
* <<esql-search-functions>>: Complete reference for all search functions | ||
* <<esql-limitations-full-text-search, Limitations>>: Current limitations for search in ES|QL | ||
|
||
[[esql-for-search-concepts]] | ||
===== Background concepts | ||
|
||
* <<analysis>>: Learn how text is processed for full-text search | ||
* <<semantic-search>>: Get an overview of semantic search in {es} | ||
* <<query-filter-context>>: Understand the difference between query and filter contexts in {es} | ||
|
||
[[esql-for-search-blogs]] | ||
===== Related blog posts | ||
|
||
// TODO* https://www.elastic.co/blog/esql-you-know-for-search-scoring-semantic-search[ES|QL, you know for Search]: Introducing scoring and semantic search | ||
* https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL]: Overview of text filtering capabilities |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.