Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
191 changes: 191 additions & 0 deletions docs/reference/esql/esql-for-search.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
[[esql-for-search]]
=== Search with {esql}

This page provides an overview of how to use {esql} for search use cases.

// [TIP]
// ====
// Prefer to get started with a hands-on tutorial? Check out <<esql-search-tutorial>>.
// ====

The following table summarizes the key search features available in {esql} and when they were introduced.

[cols="1,1,2", options="header"]
|===
|Feature |Available since |Description

|<<esql-search-functions,Full text search functions>>
|8.17
|Perform basic text searches with <<esql-match, match function>> and <<esql-search-operators,match operator (`:`)>>

|<<esql-for-search-query-string,Query string function>>
|8.17
|Execute complex queries with <<esql-qstr,`qstr`>> using Query String syntax

|<<esql-for-search-scoring,Relevance scoring>>
|8.18/9.0
|Calculate and sort by relevance with `METADATA _score`

|Enhanced match options
|8.18/9.0
|Configure text searches with additional parameters for the `match` function

|<<esql-for-search-kql,Kibana Query Language>>
|8.18/9.0
|Use Kibana Query Language with <<esql-kql,`kql`>> function

|<<esql-for-search-semantic,Semantic search>>
|8.18/9.0
|Perform semantic searches on `semantic_text` field types

|<<esql-for-search-hybrid,Hybrid search>>
|8.18/9.0
|Combine lexical and semantic search approaches with custom weights
|===

[[esql-for-search-full-text]]
==== Full text search

[[esql-for-searc-match-functio-operator]]
===== Match function and operator

ES|QL offers two syntax options for `match`, which replicate the functionality of <<query-dsl-match-query, `match`>> queries in Query DSL.

Use the compact operator syntax (`:`) for simple text matching with default parameters.

[source,esql]
----
FROM logs | WHERE match(message, "connection error")
----

Use the `match()` function syntax when you need to pass additional parameters:

[source,esql]
----
FROM products | WHERE match(name, "laptop", { "boost": 2.0 })
----

[TIP]
====
See <<match-field-params,Match field parameters>> for more advanced options using match.
====

[IMPORTANT]
====
These queries match documents but don't automatically sort by relevance. To get the most relevant results first, you need to use `METADATA _score` and sort by score. See <<esql-for-search-scoring,Relevance scoring>> for more information.
====

[[esql-for-search-query-string]]
===== Query string function (`QSTR`)

The <<esql-qstr,`qstr` function>> provides the same functionality as the Query DSL's `query_string` query. This is for advanced use cases, such as wildcard searches, searches across multiple fields, and more.

[source,esql]
----
FROM articles METADATA _score
| WHERE QSTR("(new york city) OR (big apple)")
| SORT _score DESC
| LIMIT 10
----

For complete details, refer to the <<query-dsl-query-string-query, Query DSL `query_string` docs>>.

[[esql-for-search-scoring]]
==== Relevance scoring

To get the most relevant results first, you need to retrieve and sort by relevance score:

[source,esql]
----
FROM books METADATA _score
| WHERE match(title, "Shakespeare") OR match(plot, "Shakespeare")
| SORT _score DESC
----

When working with relevance scoring in ES|QL, it's important to understand how `_score` works:

* If you don't include `METADATA _score` in your query, this only performs filtering operations with no relevance calculation.
* When you include `METADATA _score`, search functions included in `WHERE` conditions contribute to the relevance score.
* Filtering operations (like range conditions and exact matches) don't affect the score.
* Including `METADATA _score` doesn't automatically sort your results by relevance. You must explicitly use `SORT _score DESC` or `SORT _score ASC` to order your results by relevance.

[[esql-for-search-semantic]]
==== Semantic search

You can perform semantic searches over <<semantic-text, `semantic_text`>> field types using the same match syntax as full-text search.

This example uses the match operator `:``:

[source,esql]
----
FROM articles METADATA _score
| WHERE semantic_content:"What are the impacts of climate change on agriculture?"
| SORT _score DESC
----

This example uses the match function:

[source,esql]
----
FROM articles METADATA _score
| WHERE match(semantic_content, "What are the impacts of climate change on agriculture?")
| SORT _score DESC
----

[[esql-for-search-hybrid]]
==== Hybrid search

Combine traditional and semantic search with custom weights:

[source,esql]
----
FROM books METADATA _score
| WHERE match(semantic_title, "fantasy adventure", { "boost": 0.75 })
OR match(title, "fantasy adventure", { "boost": 0.25 })
| SORT _score DESC
----

[[esql-for-search-kql]]
==== Kibana Query Language (KQL)

Use the <<esql-kql,KQL function>> to use the {kibana-ref}/kuery-query.html[Kibana Query Language] in your ES|QL queries:

[source,esql]
----
FROM logs*
| WHERE KQL("http.request.method:GET AND agent.type:filebeat")
----

The `kql` function is useful when transitioning queries from Kibana's Discover, Dashboard, or other interfaces that use KQL. This will allow you to gradually migrate queries to ES|QL without needing to rewrite them all at once.

[[esql-for-search-limitations]]
==== Limitations

Refer to <<esql-limitations-full-text-search, {esql} limitations>> for a list of known limitations.

[[esql-for-search-next-steps]]
==== Next steps

[[esql-for-search-tutorials]]
===== Tutorials and how-to guides

// TODO * <<esql-search-tutorial>>: Hands-on tutorial for getting started with search tools in {esql}
* <<semantic-search-semantic-text>>: Learn how to use the `semantic_text` field type

[[esql-for-search-reference]]
===== Technical reference

* <<esql-search-functions>>: Complete reference for all search functions
* <<esql-limitations-full-text-search, Limitations>>: Current limitations for search in ES|QL

[[esql-for-search-concepts]]
===== Background concepts

* <<analysis>>: How text is processed for full-text search
* <<semantic-search>>: Overview of semantic search in Elasticsearch

[[esql-for-search-blogs]]
===== Related blog posts

// TODO* https://www.elastic.co/blog/esql-you-know-for-search-scoring-semantic-search[ES|QL, you know for Search] - Introducing scoring and semantic search
* https://www.elastic.co/blog/introducing-full-text-filtering-with-esql[Introducing full text filtering in ES|QL] - Overview of text filtering capabilities
7 changes: 5 additions & 2 deletions docs/reference/esql/esql-language.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@ Detailed reference documentation for the {esql} language:
* <<esql-functions>>
* <<esql-metadata-fields>>
* <<esql-multivalued-fields>>
* <<esql-enrich-data>>
* <<esql-for-search>>
* <<esql-lookup-join>>

* <<esql-enrich-data>>
* <<esql-process-data-with-dissect-and-grok>>
* <<esql-implicit-casting>>
* <<esql-time-spans>>
Expand All @@ -22,8 +24,9 @@ include::esql-commands.asciidoc[]
include::esql-functions-operators.asciidoc[]
include::metadata-fields.asciidoc[]
include::multivalued-fields.asciidoc[]
include::esql-for-search.asciidoc[]
include::esql-process-data-with-dissect-grok.asciidoc[]
include::esql-enrich-data.asciidoc[]
include::esql-lookup-join.asciidoc[]
include::esql-enrich-data.asciidoc[]
include::implicit-casting.asciidoc[]
include::time-spans.asciidoc[]