|
| 1 | +--- |
| 2 | +applies_to: |
| 3 | + stack: preview |
| 4 | + serverless: preview |
| 5 | +--- |
| 6 | + |
| 7 | +% ℹ️ 8.x version of this doc lives in elasticsearch repo |
| 8 | +% https://github.com/elastic/elasticsearch/blob/8.x/docs/reference/esql/esql-for-search.asciidoc |
| 9 | + |
| 10 | +# {{esql}} for search [esql-for-search] |
| 11 | + |
| 12 | +This page provides an overview of how to use {{esql}} for search use cases. |
| 13 | + |
| 14 | +::::{tip} |
| 15 | +Prefer to get started with a hands-on tutorial? Check out [Search and filter with {{esql}}](esql-search-tutorial.md). |
| 16 | +:::: |
| 17 | + |
| 18 | + |
| 19 | +## Overview |
| 20 | + |
| 21 | +The following table summarizes the key search features available in [{{esql}}](/explore-analyze/query-filter/languages/esql.md) and when they were introduced. |
| 22 | + |
| 23 | +| Feature | Available since | Description | |
| 24 | +|---------|----------------|-------------| |
| 25 | +| [Full text search functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-functions) | 8.17 | Perform basic text searches with `MATCH` function or match operator (`:`) | |
| 26 | +| [Query string function](#esql-for-search-query-string) | 8.17 | Execute complex queries with `QSTR` using Query String syntax | |
| 27 | +| [Relevance scoring](#esql-for-search-scoring) | 8.18/9.0 | Calculate and sort by relevance with `METADATA _score` | |
| 28 | +| Enhanced match options | 8.18/9.0 | Configure text searches with additional parameters for the `MATCH` function | |
| 29 | +| [Kibana Query Language](#esql-for-search-kql) | 8.18/9.0 | Use Kibana Query Language with the `KQL` function | |
| 30 | +| [Semantic search](#esql-for-search-semantic) | 8.18/9.0 | Perform semantic searches on `semantic_text` field types | |
| 31 | +| [Hybrid search](#esql-for-search-hybrid) | 8.18/9.0 | Combine lexical and semantic search approaches with custom weights | |
| 32 | + |
| 33 | +## Filtering vs. searching [esql-filtering-vs-searching] |
| 34 | + |
| 35 | +{{esql}} can be used for both simple filtering and relevance-based searching: |
| 36 | + |
| 37 | +* **Filtering** removes non-matching documents without calculating relevance scores |
| 38 | +* **Searching** both filters documents and ranks them by how well they match the query |
| 39 | + |
| 40 | +Note that filtering is faster than searching, because it doesn't require score calculations. |
| 41 | + |
| 42 | +### Relevance scoring [esql-for-search-scoring] |
| 43 | + |
| 44 | +To get the most relevant results first, you need to use `METADATA _score` and sort by score. For example: |
| 45 | + |
| 46 | +```esql |
| 47 | +FROM books METADATA _score |
| 48 | +| WHERE match(title, "Shakespeare") OR match(plot, "Shakespeare") |
| 49 | +| SORT _score DESC |
| 50 | +``` |
| 51 | + |
| 52 | +### How `_score` works [esql-for-search-how-scoring-works] |
| 53 | + |
| 54 | +When working with relevance scoring in {{esql}}: |
| 55 | + |
| 56 | +* If you don't include `METADATA _score` in your query, this only performs filtering operations with no relevance calculation. |
| 57 | +* When you include `METADATA _score`, any search function included in `WHERE` conditions contribute to the relevance score. This means that every occurrence of `MATCH`, `QSTR` and `KQL` will affect the score. |
| 58 | +* Filtering operations that are not search functions, like range conditions and exact matches, don't affect the score. |
| 59 | +* Including `METADATA _score` doesn't automatically sort your results by relevance. You must explicitly use `SORT _score DESC` or `SORT _score ASC` to order your results by relevance. |
| 60 | + |
| 61 | +## Full text search [esql-for-search-full-text] |
| 62 | + |
| 63 | +### `MATCH` function and operator [esql-for-search-match-function-operator] |
| 64 | + |
| 65 | +{{esql}} offers two syntax options for `match`, which replicate the functionality of [match](elasticsearch://reference/query-languages/query-dsl/query-dsl-match-query.md) queries in Query DSL. |
| 66 | + |
| 67 | +Use the compact operator syntax (`:`) for simple text matching with default parameters. |
| 68 | + |
| 69 | +```esql |
| 70 | +FROM logs | WHERE message: "connection error" |
| 71 | +``` |
| 72 | + |
| 73 | +Use the `match()` function syntax when you need to pass additional parameters: |
| 74 | + |
| 75 | +```esql |
| 76 | +FROM products | WHERE match(name, "laptop", { "boost": 2.0 }) |
| 77 | +``` |
| 78 | + |
| 79 | +These full-text functions address several key limitations that existed for text filtering in {{esql}}: |
| 80 | + |
| 81 | +* They work directly on multivalued fields, returning results when any value in a multivalued field matches the query |
| 82 | +* They leverage analyzers, ensuring the query is analyzed with the same process as the indexed data (enabling case-insensitive matching, ASCII folding, stopword removal, and synonym support) |
| 83 | +* They are highly performant, using Lucene index structures rather than pattern matching or regular expressions to locate terms in your data |
| 84 | + |
| 85 | +Refer to this blog for more context: [Introducing full text filtering in {{esql}}](https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr). |
| 86 | + |
| 87 | +::::{tip} |
| 88 | +See [Match field parameters](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-match) for more advanced options using match. |
| 89 | +:::: |
| 90 | + |
| 91 | +::::{important} |
| 92 | +These queries match documents but don't automatically sort by relevance. To get the most relevant results first, you need to use `METADATA _score` and sort by score. See [Relevance scoring](#esql-for-search-scoring) for more information. |
| 93 | +:::: |
| 94 | + |
| 95 | +### Query string (`QSTR`) function [esql-for-search-query-string] |
| 96 | + |
| 97 | +The [`qstr` function](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-qstr) provides the same functionality as the Query DSL's `query_string` query. This is for advanced use cases, such as wildcard searches, searches across multiple fields, and more. |
| 98 | + |
| 99 | +```esql |
| 100 | +FROM articles METADATA _score |
| 101 | +| WHERE QSTR("(new york city) OR (big apple)") |
| 102 | +| SORT _score DESC |
| 103 | +| LIMIT 10 |
| 104 | +``` |
| 105 | + |
| 106 | +For complete details, refer to the [Query DSL `query_string` docs](elasticsearch://reference/query-languages/query-dsl/query-dsl-query-string-query.md). |
| 107 | + |
| 108 | +### `KQL` function [esql-for-search-kql] |
| 109 | + |
| 110 | +Use the [KQL function](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-kql) to use the [Kibana Query Language](/explore-analyze/query-filter/languages/kql.md) in your {{esql}} queries: |
| 111 | + |
| 112 | +```esql |
| 113 | +FROM logs* |
| 114 | +| WHERE KQL("http.request.method:GET AND agent.type:filebeat") |
| 115 | +``` |
| 116 | + |
| 117 | +The `kql` function is useful when transitioning queries from Kibana's Discover, Dashboard, or other interfaces that use KQL. This will allow you to gradually migrate queries to {{esql}} without needing to rewrite them all at once. |
| 118 | + |
| 119 | +## Semantic search [esql-for-search-semantic] |
| 120 | + |
| 121 | +You can perform semantic searches over [`semantic_text`](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md) field types using the same match syntax as full-text search. |
| 122 | + |
| 123 | +This example uses the match operator `:`: |
| 124 | + |
| 125 | +```esql |
| 126 | +FROM articles METADATA _score |
| 127 | +| WHERE semantic_content: "What are the impacts of climate change on agriculture?" |
| 128 | +| SORT _score DESC |
| 129 | +``` |
| 130 | + |
| 131 | +This example uses the match function: |
| 132 | + |
| 133 | +```esql |
| 134 | +FROM articles METADATA _score |
| 135 | +| WHERE match(semantic_content, "What are the impacts of climate change on agriculture?") |
| 136 | +| SORT _score DESC |
| 137 | +``` |
| 138 | + |
| 139 | +## Hybrid search [esql-for-search-hybrid] |
| 140 | + |
| 141 | +[Hybrid search](/solutions/search/hybrid-search.md) combines lexical and semantic search with custom weights: |
| 142 | + |
| 143 | +```esql |
| 144 | +FROM books METADATA _score |
| 145 | +| WHERE match(semantic_title, "fantasy adventure", { "boost": 0.75 }) |
| 146 | + OR match(title, "fantasy adventure", { "boost": 0.25 }) |
| 147 | +| SORT _score DESC |
| 148 | +``` |
| 149 | + |
| 150 | +## Limitations [esql-for-search-limitations] |
| 151 | + |
| 152 | +Refer to [{{esql}} limitations](elasticsearch://reference/query-languages/esql/limitations.md#esql-limitations-full-text-search) for a list of known limitations. |
| 153 | + |
| 154 | +## Next steps [esql-for-search-next-steps] |
| 155 | + |
| 156 | +### Tutorials and how-to guides [esql-for-search-tutorials] |
| 157 | + |
| 158 | +- [Search and filter with {{esql}}](esql-search-tutorial.md): Hands-on tutorial for getting started with search tools in {esql} |
| 159 | +- [Semantic search with semantic_text](semantic-search/semantic-search-semantic-text.md): Learn how to use the `semantic_text` field type |
| 160 | + |
| 161 | +### Technical reference [esql-for-search-reference] |
| 162 | + |
| 163 | +- [Search functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-functions): Complete reference for all search functions |
| 164 | +- [Limitations](elasticsearch://reference/query-languages/esql/limitations.md#esql-limitations-full-text-search): Current limitations for search in {{esql}} |
| 165 | + |
| 166 | +### Background concepts [esql-for-search-concepts] |
| 167 | + |
| 168 | +- [Analysis](/manage-data/data-store/text-analysis.md): Learn how text is processed for full-text search |
| 169 | +- [Semantic search](semantic-search.md): Get an overview of semantic search in Elasticsearch |
| 170 | +- [Query vs filter context](elasticsearch://reference/query-languages/query-dsl/query-filter-context.md): Understand the difference between query and filter contexts in Elasticsearch |
| 171 | + |
| 172 | +### Related blog posts [esql-for-search-blogs] |
| 173 | + |
| 174 | +% TODO* https://www.elastic.co/blog/esql-you-know-for-search-scoring-semantic-search[{{esql}}, you know for Search]: Introducing scoring and semantic search |
| 175 | +- [Introducing full text filtering in {{esql}}](https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr): Overview of {{esql}}'s text filtering capabilities |
0 commit comments