diff --git a/docs/reference/esql/esql-for-search.asciidoc b/docs/reference/esql/esql-for-search.asciidoc new file mode 100644 index 0000000000000..3e6869298b691 --- /dev/null +++ b/docs/reference/esql/esql-for-search.asciidoc @@ -0,0 +1,215 @@ +[[esql-for-search]] +=== Using {esql} for search + +preview::[] + +This page provides an overview of how to use {esql} for search use cases. + +// [TIP] +// ==== +// Prefer to get started with a hands-on tutorial? Check out <>. +// ==== + +The following table summarizes the key search features available in {esql} and when they were introduced. + +[cols="1,1,2", options="header"] +|=== +|Feature |Available since |Description + +|<> +|8.17 +|Perform basic text searches with <> and <> + +|<> +|8.17 +|Execute complex queries with <> using Query String syntax + +|<> +|8.18/9.0 +|Calculate and sort by relevance with `METADATA _score` + +|Enhanced match options +|8.18/9.0 +|Configure text searches with additional parameters for the `match` function + +|<> +|8.18/9.0 +|Use Kibana Query Language with <> function + +|<> +|8.18/9.0 +|Perform semantic searches on `semantic_text` field types + +|<> +|8.18/9.0 +|Combine lexical and semantic search approaches with custom weights +|=== + +[[esql-filtering-vs-searching]] +==== Filtering vs. searching + +{esql} can be used for both simple filtering and relevance-based searching: + +* **Filtering** removes non-matching documents without calculating relevance scores +* **Searching** both filters documents and ranks them by how well they match the query + +Note that filtering is faster than searching, because it doesn't require score calculations. + +[[esql-for-search-scoring]] +===== Relevance scoring + +To get the most relevant results first, you need to use `METADATA _score` and sort by score. For example: + +[source,esql] +---- +FROM books METADATA _score +| WHERE match(title, "Shakespeare") OR match(plot, "Shakespeare") +| SORT _score DESC +---- + +[[esql-for-search-how-scoring-works]] +===== How `_score` works + +When working with relevance scoring in ES|QL: + +* If you don't include `METADATA _score` in your query, this only performs filtering operations with no relevance calculation. +* When you include `METADATA _score`, any search function included in `WHERE` conditions contribute to the relevance score. This means that every occurrence of `MATCH`, `QSTR` and `KQL` will affect the score. +* Filtering operations that are not search functions, like range conditions and exact matches, don't affect the score. +* Including `METADATA _score` doesn't automatically sort your results by relevance. You must explicitly use `SORT _score DESC` or `SORT _score ASC` to order your results by relevance. + +[[esql-for-search-full-text]] +==== Full text search + +[[esql-for-search-match-function-operator]] +===== Match function and operator + +ES|QL offers two syntax options for `match`, which replicate the functionality of <> queries in Query DSL. + +Use the compact operator syntax (`:`) for simple text matching with default parameters. + +[source,esql] +---- +FROM logs | WHERE match(message, "connection error") +---- + +Use the `match()` function syntax when you need to pass additional parameters: + +[source,esql] +---- +FROM products | WHERE match(name, "laptop", { "boost": 2.0 }) +---- + +These full-text functions address several key limitations that existed for text filtering in {esql}: + +* They work directly on multivalued fields, returning results when any value in a multivalued field matches the query +* They leverage analyzers, ensuring the query is analyzed with the same process as the indexed data (enabling case-insensitive matching, ASCII folding, stopword removal, and synonym support) +* They are highly performant, using Lucene index structures rather than pattern matching or regular expressions to locate terms in your data + +Refer to this blog for more context: https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL]. + +[TIP] +==== +See <> for more advanced options using match. +==== + +[IMPORTANT] +==== +These queries match documents but don't automatically sort by relevance. To get the most relevant results first, you need to use `METADATA _score` and sort by score. See <> for more information. +==== + +[[esql-for-search-query-string]] +===== Query string function (`QSTR`) + +The <> provides the same functionality as the Query DSL's `query_string` query. This is for advanced use cases, such as wildcard searches, searches across multiple fields, and more. + +[source,esql] +---- +FROM articles METADATA _score +| WHERE QSTR("(new york city) OR (big apple)") +| SORT _score DESC +| LIMIT 10 +---- + +For complete details, refer to the <>. + +[[esql-for-search-kql]] +===== Kibana Query Language function (`KQL`) + +Use the <> to use the {kibana-ref}/kuery-query.html[Kibana Query Language] in your ES|QL queries: + +[source,esql] +---- +FROM logs* +| WHERE KQL("http.request.method:GET AND agent.type:filebeat") +---- + +The `kql` function is useful when transitioning queries from Kibana's Discover, Dashboard, or other interfaces that use KQL. This will allow you to gradually migrate queries to ES|QL without needing to rewrite them all at once. + +[[esql-for-search-semantic]] +==== Semantic search + +You can perform semantic searches over <> field types using the same match syntax as full-text search. + +This example uses the match operator `:`: + +[source,esql] +---- +FROM articles METADATA _score +| WHERE semantic_content:"What are the impacts of climate change on agriculture?" +| SORT _score DESC +---- + +This example uses the match function: + +[source,esql] +---- +FROM articles METADATA _score +| WHERE match(semantic_content, "What are the impacts of climate change on agriculture?") +| SORT _score DESC +---- + +[[esql-for-search-hybrid]] +==== Hybrid search + +Combine traditional and semantic search with custom weights: + +[source,esql] +---- +FROM books METADATA _score +| WHERE match(semantic_title, "fantasy adventure", { "boost": 0.75 }) + OR match(title, "fantasy adventure", { "boost": 0.25 }) +| SORT _score DESC +---- + +[[esql-for-search-limitations]] +==== Limitations + +Refer to <> for a list of known limitations. + +[[esql-for-search-next-steps]] +==== Next steps + +[[esql-for-search-tutorials]] +===== Tutorials and how-to guides + +// TODO * <>: Hands-on tutorial for getting started with search tools in {esql} +* <>: Learn how to use the `semantic_text` field type + +[[esql-for-search-reference]] +===== Technical reference + +* <>: Complete reference for all search functions +* <>: Current limitations for search in ES|QL + +[[esql-for-search-concepts]] +===== Background concepts + +* <>: Learn how text is processed for full-text search +* <>: Get an overview of semantic search in {es} +* <>: Understand the difference between query and filter contexts in {es} + +[[esql-for-search-blogs]] +===== Related blog posts + +// TODO* https://www.elastic.co/blog/esql-you-know-for-search-scoring-semantic-search[ES|QL, you know for Search]: Introducing scoring and semantic search +* https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL]: Overview of text filtering capabilities diff --git a/docs/reference/esql/esql-get-started.asciidoc b/docs/reference/esql/esql-get-started.asciidoc index b7928898a3bbb..6451ed1909201 100644 --- a/docs/reference/esql/esql-get-started.asciidoc +++ b/docs/reference/esql/esql-get-started.asciidoc @@ -4,7 +4,7 @@ Getting started ++++ -This guide shows how you can use {esql} to query and aggregate your data. +This guide shows how you can use {esql} to query and aggregate your data. Refer to <> if you'd like to learn more about using {esql} for search use cases. [TIP] ==== diff --git a/docs/reference/esql/esql-language.asciidoc b/docs/reference/esql/esql-language.asciidoc index cb2d8260469f6..38960df404a37 100644 --- a/docs/reference/esql/esql-language.asciidoc +++ b/docs/reference/esql/esql-language.asciidoc @@ -11,8 +11,8 @@ Detailed reference documentation for the {esql} language: * <> * <> * <> -* <> * <> +* <> * <> * <> * <> @@ -23,7 +23,7 @@ include::esql-functions-operators.asciidoc[] include::metadata-fields.asciidoc[] include::multivalued-fields.asciidoc[] include::esql-process-data-with-dissect-grok.asciidoc[] -include::esql-enrich-data.asciidoc[] include::esql-lookup-join.asciidoc[] +include::esql-enrich-data.asciidoc[] include::implicit-casting.asciidoc[] include::time-spans.asciidoc[] diff --git a/docs/reference/esql/esql-using.asciidoc b/docs/reference/esql/esql-using.asciidoc index d2e18bf1b91a3..a7b7ad1998ab9 100644 --- a/docs/reference/esql/esql-using.asciidoc +++ b/docs/reference/esql/esql-using.asciidoc @@ -1,27 +1,33 @@ [[esql-using]] == Using {esql} +This page is an overview of the various ways you can use {esql} across different Elastic interfaces and use cases. + <>:: -Information about using the <>. +Learn how to use the <>. + +<>:: +Learn how to use {esql} for search use cases. <>:: -Using {esql} in {kib} to query and aggregate your data, create visualizations, +Learn how to use {esql} in {kib} to query and aggregate your data, create visualizations, and set up alerts. <>:: -Using {esql} in {elastic-sec} to investigate events in Timeline, create +Learn how to use {esql} in {elastic-sec} to investigate events in Timeline, create detection rules, and build {esql} queries using Elastic AI Assistant. <>:: -Using {esql} to query multiple indexes and resolve field type mismatches. +Learn how to use {esql} to query multiple indexes and resolve field type mismatches. <>:: -Using {esql} to query across multiple clusters. +Learn how to use {esql} to query across multiple clusters. <>:: -Using the <> to list and cancel {esql} queries. +Learn how to use the <> to list and cancel {esql} queries. include::esql-rest.asciidoc[] +include::esql-for-search.asciidoc[] include::esql-kibana.asciidoc[] include::esql-security-solution.asciidoc[] include::esql-multi-index.asciidoc[] diff --git a/docs/reference/esql/index.asciidoc b/docs/reference/esql/index.asciidoc index 54627a6de3c62..7c8c793d483bb 100644 --- a/docs/reference/esql/index.asciidoc +++ b/docs/reference/esql/index.asciidoc @@ -22,21 +22,7 @@ a series of operations, where the output of one operation becomes the input for the next, enabling complex data transformations and analysis. [discrete] -=== The {esql} Compute Engine - -{esql} is more than a language: it represents a significant investment in new -compute capabilities within {es}. To achieve both the functional and performance -requirements for {esql}, it was necessary to build an entirely new compute -architecture. {esql} search, aggregation, and transformation functions are -directly executed within Elasticsearch itself. Query expressions are not -transpiled to Query DSL for execution. This approach allows {esql} to be -extremely performant and versatile. - -The new {esql} execution engine was designed with performance in mind — it -operates on blocks at a time instead of per row, targets vectorization and cache -locality, and embraces specialization and multi-threading. It is a separate -component from the existing Elasticsearch aggregation framework with different -performance characteristics. +=== Documentation organization The {esql} documentation is organized in these sections: @@ -45,16 +31,24 @@ A tutorial to help you get started with {esql}. <>:: -Reference documentation for the <>, -<>, and <>. Information about working with <> and <>. And guidance for -<> and <>. +Reference documentation for the <>: + +* Reference for <>, and <> +* How to work with <> and <> +* How to work with +<>, <>, and <> <>:: -An overview of using the <>, <>, -<>, <>, and <>. +An overview of: +* <> +* <> +* <> +* <> +* <> +* <> <>:: The current limitations of {esql}. @@ -62,6 +56,8 @@ The current limitations of {esql}. <>:: A few examples of what you can do with {esql}. + + include::esql-get-started.asciidoc[] include::esql-language.asciidoc[] @@ -74,3 +70,20 @@ include::esql-examples.asciidoc[] :esql-tests!: :esql-specs!: + +[discrete] +=== The {esql} Compute Engine + +{esql} is more than a language: it represents a significant investment in new +compute capabilities within {es}. To achieve both the functional and performance +requirements for {esql}, it was necessary to build an entirely new compute +architecture. {esql} search, aggregation, and transformation functions are +directly executed within Elasticsearch itself. Query expressions are not +transpiled to Query DSL for execution. This approach allows {esql} to be +extremely performant and versatile. + +The new {esql} execution engine was designed with performance in mind — it +operates on blocks at a time instead of per row, targets vectorization and cache +locality, and embraces specialization and multi-threading. It is a separate +component from the existing Elasticsearch aggregation framework with different +performance characteristics. \ No newline at end of file