diff --git a/explore-analyze/discover/try-esql.md b/explore-analyze/discover/try-esql.md index d758239521..18145db85d 100644 --- a/explore-analyze/discover/try-esql.md +++ b/explore-analyze/discover/try-esql.md @@ -135,7 +135,7 @@ FROM kibana_sample_data_ecommerce ### ES|QL and LOOKUP JOINs -The ES|QL editor supports [`LOOKUP JOIN`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-lookup-join) commands and suggests lookup mode indices and join condition fields. +The ES|QL editor supports [`LOOKUP JOIN`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) commands and suggests lookup mode indices and join condition fields. ![Using the LOOKUP JOIN command to autocomplete an ES|QL query](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte43a30a93241d650/67c23670045f5839e5bfd1e4/lookup-join-demo.gif) diff --git a/explore-analyze/geospatial-analysis.md b/explore-analyze/geospatial-analysis.md index c75063839e..1a5fc2782a 100644 --- a/explore-analyze/geospatial-analysis.md +++ b/explore-analyze/geospatial-analysis.md @@ -37,13 +37,13 @@ Data is often messy and incomplete. [Ingest pipelines](../manage-data/ingest/tra ## ES|QL [esql-query] -[ES|QL](query-filter/languages/esql.md) has support for [Geospatial Search](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-spatial-functions) functions, enabling efficient index searching for documents that intersect with, are within, are contained by, or are disjoint from a query geometry. In addition, the `ST_DISTANCE` function calculates the distance between two points. +[ES|QL](query-filter/languages/esql.md) has support for [Geospatial Search](elasticsearch://reference/query-languages/esql/functions-operators/spatial-functions.md) functions, enabling efficient index searching for documents that intersect with, are within, are contained by, or are disjoint from a query geometry. In addition, the `ST_DISTANCE` function calculates the distance between two points. -* [`ST_INTERSECTS`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-st_intersects) -* [`ST_DISJOINT`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-st_disjoint) -* [`ST_CONTAINS`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-st_contains) -* [`ST_WITHIN`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-st_within) -* [`ST_DISTANCE`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-st_distance) +* [`ST_INTERSECTS`](elasticsearch://reference/query-languages/esql/functions-operators/spatial-functions.md#esql-st_intersects) +* [`ST_DISJOINT`](elasticsearch://reference/query-languages/esql/functions-operators/spatial-functions.md#esql-st_disjoint) +* [`ST_CONTAINS`](elasticsearch://reference/query-languages/esql/functions-operators/spatial-functions.md#esql-st_contains) +* [`ST_WITHIN`](elasticsearch://reference/query-languages/esql/functions-operators/spatial-functions.md#esql-st_within) +* [`ST_DISTANCE`](elasticsearch://reference/query-languages/esql/functions-operators/spatial-functions.md#esql-st_distance) ## Aggregate [geospatial-aggregate] diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index 3b68c6b7b5..812ee7a35e 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -362,7 +362,7 @@ Which returns: ## Enrich across clusters [ccq-enrich] -Enrich in {{esql}} across clusters operates similarly to [local enrich](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-enrich). If the enrich policy and its enrich indices are consistent across all clusters, simply write the enrich command as you would without remote clusters. In this default mode, {{esql}} can execute the enrich command on either the local cluster or the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query result. +Enrich in {{esql}} across clusters operates similarly to [local enrich](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-enrich). If the enrich policy and its enrich indices are consistent across all clusters, simply write the enrich command as you would without remote clusters. In this default mode, {{esql}} can execute the enrich command on either the local cluster or the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query result. ::::{tip} Enrich in {{esql}} across clusters using the API key based security model was introduced in version **8.15.0**. Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to use the new required permissions. Refer to the example in the [API key authentication](#esql-ccs-security-model-api-key) section. @@ -418,7 +418,7 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` -A `_remote` enrich cannot be executed after a [stats](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-stats-by) command. The following example would result in an error: +A `_remote` enrich cannot be executed after a [stats](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-stats-by) command. The following example would result in an error: ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 diff --git a/explore-analyze/query-filter/languages/esql-getting-started.md b/explore-analyze/query-filter/languages/esql-getting-started.md index d936b4b43d..2b99318930 100644 --- a/explore-analyze/query-filter/languages/esql-getting-started.md +++ b/explore-analyze/query-filter/languages/esql-getting-started.md @@ -114,13 +114,13 @@ You can adjust the editor’s height by dragging its bottom border to your likin ## Your first {{esql}} query [esql-getting-started-first-query] -Each {{esql}} query starts with a [source command](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-source-commands). A source command produces a table, typically with data from {{es}}. +Each {{esql}} query starts with a [source command](elasticsearch://reference/query-languages/esql/commands/source-commands.md). A source command produces a table, typically with data from {{es}}. :::{image} /explore-analyze/images/elasticsearch-reference-source-command.svg :alt: A source command producing a table from {{es}} ::: -The [`FROM`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-from) source command returns a table with documents from a data stream, index, or alias. Each row in the resulting table represents a document. This query returns up to 1000 documents from the `sample_data` index: +The [`FROM`](elasticsearch://reference/query-languages/esql/commands/source-commands.md#esql-from) source command returns a table with documents from a data stream, index, or alias. Each row in the resulting table represents a document. This query returns up to 1000 documents from the `sample_data` index: ```esql FROM sample_data @@ -141,13 +141,13 @@ from sample_data ## Processing commands [esql-getting-started-limit] -A source command can be followed by one or more [processing commands](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-processing-commands), separated by a pipe character: `|`. Processing commands change an input table by adding, removing, or changing rows and columns. Processing commands can perform filtering, projection, aggregation, and more. +A source command can be followed by one or more [processing commands](elasticsearch://reference/query-languages/esql/commands/processing-commands.md), separated by a pipe character: `|`. Processing commands change an input table by adding, removing, or changing rows and columns. Processing commands can perform filtering, projection, aggregation, and more. :::{image} /explore-analyze/images/elasticsearch-reference-esql-limit.png :alt: A processing command changing an input table ::: -For example, you can use the [`LIMIT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-limit) command to limit the number of rows that are returned, up to a maximum of 10,000 rows: +For example, you can use the [`LIMIT`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-limit) command to limit the number of rows that are returned, up to a maximum of 10,000 rows: ```esql FROM sample_data @@ -171,7 +171,7 @@ FROM sample_data | LIMIT 3 :alt: A processing command sorting an input table ::: -Another processing command is the [`SORT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-sort) command. By default, the rows returned by `FROM` don’t have a defined sort order. Use the `SORT` command to sort rows on one or more columns: +Another processing command is the [`SORT`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-sort) command. By default, the rows returned by `FROM` don’t have a defined sort order. Use the `SORT` command to sort rows on one or more columns: ```esql FROM sample_data @@ -181,14 +181,14 @@ FROM sample_data ### Query the data [esql-getting-started-where] -Use the [`WHERE`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-where) command to query the data. For example, to find all events with a duration longer than 5ms: +Use the [`WHERE`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-where) command to query the data. For example, to find all events with a duration longer than 5ms: ```esql FROM sample_data | WHERE event_duration > 5000000 ``` -`WHERE` supports several [operators](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-operators). For example, you can use [`LIKE`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-like) to run a wildcard query against the `message` column: +`WHERE` supports several [operators](elasticsearch://reference/query-languages/esql/functions-operators/operators.md). For example, you can use [`LIKE`](elasticsearch://reference/query-languages/esql/functions-operators/operators.md#esql-like) to run a wildcard query against the `message` column: ```esql FROM sample_data @@ -198,7 +198,7 @@ FROM sample_data ### More processing commands [esql-getting-started-more-commands] -There are many other processing commands, like [`KEEP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-keep) and [`DROP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-drop) to keep or drop columns, [`ENRICH`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-enrich) to enrich a table with data from indices in {{es}}, and [`DISSECT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-dissect) and [`GROK`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-grok) to process data. Refer to [Processing commands](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-processing-commands) for an overview of all processing commands. +There are many other processing commands, like [`KEEP`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-keep) and [`DROP`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-drop) to keep or drop columns, [`ENRICH`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-enrich) to enrich a table with data from indices in {{es}}, and [`DISSECT`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-dissect) and [`GROK`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-grok) to process data. Refer to [Processing commands](elasticsearch://reference/query-languages/esql/commands/processing-commands.md) for an overview of all processing commands. ## Chain processing commands [esql-getting-started-chaining] @@ -225,14 +225,14 @@ The order of processing commands is important. First limiting the result set to ## Compute values [esql-getting-started-eval] -Use the [`EVAL`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-eval) command to append columns to a table, with calculated values. For example, the following query appends a `duration_ms` column. The values in the column are computed by dividing `event_duration` by 1,000,000. In other words: `event_duration` converted from nanoseconds to milliseconds. +Use the [`EVAL`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-eval) command to append columns to a table, with calculated values. For example, the following query appends a `duration_ms` column. The values in the column are computed by dividing `event_duration` by 1,000,000. In other words: `event_duration` converted from nanoseconds to milliseconds. ```esql FROM sample_data | EVAL duration_ms = event_duration/1000000.0 ``` -`EVAL` supports several [functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-functions). For example, to round a number to the closest number with the specified number of digits, use the [`ROUND`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-round) function: +`EVAL` supports several [functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-functions). For example, to round a number to the closest number with the specified number of digits, use the [`ROUND`](elasticsearch://reference/query-languages/esql/functions-operators/math-functions.md#esql-round) function: ```esql FROM sample_data @@ -242,7 +242,7 @@ FROM sample_data ## Calculate statistics [esql-getting-started-stats] -{{esql}} can not only be used to query your data, you can also use it to aggregate your data. Use the [`STATS`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-stats-by) command to calculate statistics. For example, the median duration: +{{esql}} can not only be used to query your data, you can also use it to aggregate your data. Use the [`STATS`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-stats-by) command to calculate statistics. For example, the median duration: ```esql FROM sample_data @@ -286,9 +286,9 @@ FROM sample_data ## Create a histogram [esql-getting-started-histogram] -To track statistics over time, {{esql}} enables you to create histograms using the [`BUCKET`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-bucket) function. `BUCKET` creates human-friendly bucket sizes and returns a value for each row that corresponds to the resulting bucket the row falls into. +To track statistics over time, {{esql}} enables you to create histograms using the [`BUCKET`](elasticsearch://reference/query-languages/esql/functions-operators/grouping-functions.md#esql-bucket) function. `BUCKET` creates human-friendly bucket sizes and returns a value for each row that corresponds to the resulting bucket the row falls into. -Combine `BUCKET` with [`STATS`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-stats-by) to create a histogram. For example, to count the number of events per hour: +Combine `BUCKET` with [`STATS`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-stats-by) to create a histogram. For example, to count the number of events per hour: ```esql FROM sample_data @@ -306,7 +306,7 @@ FROM sample_data ## Enrich data [esql-getting-started-enrich] -{{esql}} enables you to [enrich](elasticsearch://reference/query-languages/esql/esql-enrich-data.md) a table with data from indices in {{es}}, using the [`ENRICH`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-enrich) command. +{{esql}} enables you to [enrich](elasticsearch://reference/query-languages/esql/esql-enrich-data.md) a table with data from indices in {{es}}, using the [`ENRICH`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-enrich) command. :::{image} /explore-analyze/images/elasticsearch-reference-esql-enrich.png :alt: esql enrich @@ -396,7 +396,7 @@ Your data may contain unstructured strings that you want to [structure](elastics By extracting the IP address from these messages, you can determine which IP has accepted the most client connections. -To structure unstructured strings at query time, you can use the {{esql}} [`DISSECT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-dissect) and [`GROK`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-grok) commands. `DISSECT` works by breaking up a string using a delimiter-based pattern. `GROK` works similarly, but uses regular expressions. This makes `GROK` more powerful, but generally also slower. +To structure unstructured strings at query time, you can use the {{esql}} [`DISSECT`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-dissect) and [`GROK`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-grok) commands. `DISSECT` works by breaking up a string using a delimiter-based pattern. `GROK` works similarly, but uses regular expressions. This makes `GROK` more powerful, but generally also slower. In this case, no regular expressions are needed, as the `message` is straightforward: "Connected to ", followed by the server IP. To match this string, you can use the following `DISSECT` command: diff --git a/explore-analyze/query-filter/languages/esql-kibana.md b/explore-analyze/query-filter/languages/esql-kibana.md index 8e2deb95f0..7b9c0d8ab3 100644 --- a/explore-analyze/query-filter/languages/esql-kibana.md +++ b/explore-analyze/query-filter/languages/esql-kibana.md @@ -39,9 +39,9 @@ After switching to {{esql}} mode, the query bar shows your previous KQL or Lucen from kibana_sample_data_logs | limit 10 ``` -Every query starts with a [source command](elasticsearch://reference/query-languages/esql/esql-commands.md). In this query, the source command is [`FROM`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-from). `FROM` retrieves data from data streams, indices, or aliases. In this example, the data is retrieved from `kibana_sample_data_logs`. +Every query starts with a [source command](elasticsearch://reference/query-languages/esql/esql-commands.md). In this query, the source command is [`FROM`](elasticsearch://reference/query-languages/esql/commands/source-commands.md#esql-from). `FROM` retrieves data from data streams, indices, or aliases. In this example, the data is retrieved from `kibana_sample_data_logs`. -A source command can be followed by one or more [processing commands](elasticsearch://reference/query-languages/esql/esql-commands.md). In this query, the processing command is [`LIMIT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-limit). `LIMIT` limits the number of rows that are retrieved. +A source command can be followed by one or more [processing commands](elasticsearch://reference/query-languages/esql/esql-commands.md). In this query, the processing command is [`LIMIT`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-limit). `LIMIT` limits the number of rows that are retrieved. ::::{tip} Click the **ES|QL help** button to open the in-product reference documentation for all commands and functions or to get recommended queries that will help you get started. @@ -130,7 +130,7 @@ the 10,000 row limit only applies to the number of rows that are retrieved by th :::: -Each row shows two columns for the example query: a column with the `@timestamp` field and a column with the full document. To display specific fields from the documents, use the [`KEEP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-keep) command: +Each row shows two columns for the example query: a column with the `@timestamp` field and a column with the full document. To display specific fields from the documents, use the [`KEEP`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-keep) command: ```esql FROM kibana_sample_data_logs @@ -152,7 +152,7 @@ The maximum number of columns in Discover is 50. If a query returns more than 50 ### Sorting [_sorting] -To sort on one of the columns, click the column name you want to sort on and select the sort order. Note that this performs client-side sorting. It only sorts the rows that were retrieved by the query, which may not be the full dataset because of the (implicit) limit. To sort the full data set, use the [`SORT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-sort) command: +To sort on one of the columns, click the column name you want to sort on and select the sort order. Note that this performs client-side sorting. It only sorts the rows that were retrieved by the query, which may not be the full dataset because of the (implicit) limit. To sort the full data set, use the [`SORT`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-sort) command: ```esql FROM kibana_sample_data_logs @@ -180,7 +180,7 @@ FROM my_index | WHERE custom_timestamp >= ?_tstart AND custom_timestamp < ?_tend ``` -You can also use the `?_tstart` and `?_tend` parameters with the [`BUCKET`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-bucket) function to create auto-incrementing time buckets in {{esql}} [visualizations](#esql-kibana-visualizations). For example: +You can also use the `?_tstart` and `?_tend` parameters with the [`BUCKET`](elasticsearch://reference/query-languages/esql/functions-operators/grouping-functions.md#esql-bucket) function to create auto-incrementing time buckets in {{esql}} [visualizations](#esql-kibana-visualizations). For example: ```esql FROM kibana_sample_data_logs @@ -192,7 +192,7 @@ This example uses `50` buckets, which is the maximum number of buckets. #### WHERE command [_where_command] -You can also limit the time range using the [`WHERE`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-where) command and the [`NOW`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-now) function. For example, if the timestamp field is called `timestamp`, to query the last 15 minutes of data: +You can also limit the time range using the [`WHERE`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-where) command and the [`NOW`](elasticsearch://reference/query-languages/esql/functions-operators/date-time-functions.md#esql-now) function. For example, if the timestamp field is called `timestamp`, to query the last 15 minutes of data: ```esql FROM kibana_sample_data_logs @@ -255,7 +255,7 @@ You can also [Add dashboard controls from your ES|QL visualization's query](/exp ## Create an enrich policy [esql-kibana-enrich] -The {{esql}} [`ENRICH`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-enrich) command enables you to [enrich](elasticsearch://reference/query-languages/esql/esql-enrich-data.md) your query dataset with fields from another dataset. Before you can use `ENRICH`, you need to [create and execute an enrich policy](elasticsearch://reference/query-languages/esql/esql-enrich-data.md#esql-set-up-enrich-policy). If a policy exists, it will be suggested by auto-complete. If not, click **Click to create** to create one. +The {{esql}} [`ENRICH`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-enrich) command enables you to [enrich](elasticsearch://reference/query-languages/esql/esql-enrich-data.md) your query dataset with fields from another dataset. Before you can use `ENRICH`, you need to [create and execute an enrich policy](elasticsearch://reference/query-languages/esql/esql-enrich-data.md#esql-set-up-enrich-policy). If a policy exists, it will be suggested by auto-complete. If not, click **Click to create** to create one. :::{image} /explore-analyze/images/elasticsearch-reference-esql-kibana-enrich-autocomplete.png :alt: esql kibana enrich autocomplete @@ -301,8 +301,8 @@ You can use {{esql}} queries to create alerts. From Discover, click **Alerts** a ## Limitations [esql-kibana-limitations] -* The user interface to filter data is not enabled when Discover is in {{esql}} mode. To filter data, write a query that uses the [`WHERE`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-where) command instead. +* The user interface to filter data is not enabled when Discover is in {{esql}} mode. To filter data, write a query that uses the [`WHERE`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-where) command instead. * Discover shows no more than 10,000 rows. This limit only applies to the number of rows that are retrieved by the query and displayed in Discover. Queries and aggregations run on the full data set. * Discover shows no more than 50 columns. If a query returns more than 50 columns, Discover only shows the first 50. * CSV export from Discover shows no more than 10,000 rows. This limit only applies to the number of rows that are retrieved by the query and displayed in Discover. Queries and aggregations run on the full data set. -* Querying many indices at once without any filters can cause an error in kibana which looks like `[esql] > Unexpected error from Elasticsearch: The content length (536885793) is bigger than the maximum allowed string (536870888)`. The response from {{esql}} is too long. Use [`DROP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-drop) or [`KEEP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-keep) to limit the number of fields returned. +* Querying many indices at once without any filters can cause an error in kibana which looks like `[esql] > Unexpected error from Elasticsearch: The content length (536885793) is bigger than the maximum allowed string (536870888)`. The response from {{esql}} is too long. Use [`DROP`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-drop) or [`KEEP`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-keep) to limit the number of fields returned. diff --git a/explore-analyze/query-filter/languages/esql-multi-index.md b/explore-analyze/query-filter/languages/esql-multi-index.md index 9222e484a0..6be3cab600 100644 --- a/explore-analyze/query-filter/languages/esql-multi-index.md +++ b/explore-analyze/query-filter/languages/esql-multi-index.md @@ -113,9 +113,9 @@ This functionality is in technical preview and may be changed or removed in a fu :::: -{{esql}} has a way to handle [field type mismatches](#esql-multi-index-invalid-mapping). When the same field is mapped to multiple types in multiple indices, the type of the field is understood to be a *union* of the various types in the index mappings. As seen in the preceding examples, this *union type* cannot be used in the results, and cannot be referred to by the query — except in `KEEP`, `DROP` or when it’s passed to a type conversion function that accepts all the types in the *union* and converts the field to a single type. {{esql}} offers a suite of [type conversion functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-type-conversion-functions) to achieve this. +{{esql}} has a way to handle [field type mismatches](#esql-multi-index-invalid-mapping). When the same field is mapped to multiple types in multiple indices, the type of the field is understood to be a *union* of the various types in the index mappings. As seen in the preceding examples, this *union type* cannot be used in the results, and cannot be referred to by the query — except in `KEEP`, `DROP` or when it’s passed to a type conversion function that accepts all the types in the *union* and converts the field to a single type. {{esql}} offers a suite of [type conversion functions](elasticsearch://reference/query-languages/esql/functions-operators/type-conversion-functions.md) to achieve this. -In the above examples, the query can use a command like `EVAL client_ip = TO_IP(client_ip)` to resolve the union of `ip` and `keyword` to just `ip`. You can also use the type-conversion syntax `EVAL client_ip = client_ip::IP`. Alternatively, the query could use [`TO_STRING`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-to_string) to convert all supported types into `KEYWORD`. +In the above examples, the query can use a command like `EVAL client_ip = TO_IP(client_ip)` to resolve the union of `ip` and `keyword` to just `ip`. You can also use the type-conversion syntax `EVAL client_ip = client_ip::IP`. Alternatively, the query could use [`TO_STRING`](elasticsearch://reference/query-languages/esql/functions-operators/type-conversion-functions.md#esql-to_string) to convert all supported types into `KEYWORD`. For example, the [query](#query-unsupported) that returned `client_ip:unsupported` with `null` values can be improved using the `TO_IP` function or the equivalent `field::ip` syntax. These changes also resolve the error message. As long as the only reference to the original field is to pass it to a conversion function that resolves the type ambiguity, no error results. @@ -139,7 +139,7 @@ FROM events_* ## Index metadata [esql-multi-index-index-metadata] -It can be helpful to know the particular index from which each row is sourced. To get this information, use the [`METADATA`](elasticsearch://reference/query-languages/esql/esql-metadata-fields.md) option on the [`FROM`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-from) command. +It can be helpful to know the particular index from which each row is sourced. To get this information, use the [`METADATA`](elasticsearch://reference/query-languages/esql/esql-metadata-fields.md) option on the [`FROM`](elasticsearch://reference/query-languages/esql/commands/source-commands.md#esql-from) command. ```esql FROM events_* METADATA _index diff --git a/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md b/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md index 6909fe7423..bd4106c2aa 100644 --- a/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md +++ b/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md @@ -68,7 +68,7 @@ Once the enrich policy is created, you need to execute it using the [execute enr The *enrich index* contains documents from the policy’s source indices. Enrich indices always begin with `.enrich-*`, are read-only, and are [force merged](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge). ::::{warning} -Enrich indices should only be used by the [enrich processor](elasticsearch://reference/enrich-processor/enrich-processor.md) or the [{{esql}} `ENRICH` command](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-enrich). Avoid using enrich indices for other purposes. +Enrich indices should only be used by the [enrich processor](elasticsearch://reference/enrich-processor/enrich-processor.md) or the [{{esql}} `ENRICH` command](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-enrich). Avoid using enrich indices for other purposes. :::: diff --git a/solutions/search/esql-for-search.md b/solutions/search/esql-for-search.md index 16c6780e65..51c8d5ad42 100644 --- a/solutions/search/esql-for-search.md +++ b/solutions/search/esql-for-search.md @@ -22,7 +22,7 @@ The following table summarizes the key search features available in [{{esql}}](/ | Feature | Available since | Description | |---------|----------------|-------------| -| [Full text search functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-functions) | 8.17 | Perform basic text searches with `MATCH` function or match operator (`:`) | +| [Full text search functions](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md) | 8.17 | Perform basic text searches with `MATCH` function or match operator (`:`) | | [Query string function](#esql-for-search-query-string) | 8.17 | Execute complex queries with `QSTR` using Query String syntax | | [Relevance scoring](#esql-for-search-scoring) | 8.18/9.0 | Calculate and sort by relevance with `METADATA _score` | | Enhanced match options | 8.18/9.0 | Configure text searches with additional parameters for the `MATCH` function | @@ -85,7 +85,7 @@ These full-text functions address several key limitations that existed for text Refer to this blog for more context: [Introducing full text filtering in {{esql}}](https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr). ::::{tip} -See [Match field parameters](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-match) for more advanced options using match. +See [Match field parameters](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md#esql-match) for more advanced options using match. :::: ::::{important} @@ -94,7 +94,7 @@ These queries match documents but don't automatically sort by relevance. To get ### Query string (`QSTR`) function [esql-for-search-query-string] -The [`qstr` function](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-qstr) provides the same functionality as the Query DSL's `query_string` query. This is for advanced use cases, such as wildcard searches, searches across multiple fields, and more. +The [`qstr` function](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md#esql-qstr) provides the same functionality as the Query DSL's `query_string` query. This is for advanced use cases, such as wildcard searches, searches across multiple fields, and more. ```esql FROM articles METADATA _score @@ -107,7 +107,7 @@ For complete details, refer to the [Query DSL `query_string` docs](elasticsearch ### `KQL` function [esql-for-search-kql] -Use the [KQL function](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-kql) to use the [Kibana Query Language](/explore-analyze/query-filter/languages/kql.md) in your {{esql}} queries: +Use the [KQL function](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md#esql-kql) to use the [Kibana Query Language](/explore-analyze/query-filter/languages/kql.md) in your {{esql}} queries: ```esql FROM logs* @@ -160,7 +160,7 @@ Refer to [{{esql}} limitations](elasticsearch://reference/query-languages/esql/l ### Technical reference [esql-for-search-reference] -- [Search functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-functions): Complete reference for all search functions +- [Search functions](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md): Complete reference for all search functions - [Limitations](elasticsearch://reference/query-languages/esql/limitations.md#esql-limitations-full-text-search): Current limitations for search in {{esql}} ### Background concepts [esql-for-search-concepts] diff --git a/solutions/search/esql-search-tutorial.md b/solutions/search/esql-search-tutorial.md index 28d867be53..ee4e30bf21 100644 --- a/solutions/search/esql-search-tutorial.md +++ b/solutions/search/esql-search-tutorial.md @@ -152,12 +152,12 @@ Full-text search involves executing text-based queries across one or more docume :::{tip} {{esql}} provides two ways to perform full-text searches: -1. Full [match function](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-match) syntax: `match(field, "search terms")` -1. Compact syntax using the [match operator `:`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-operators): `field:"search terms"` +1. Full [match function](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md#esql-match) syntax: `match(field, "search terms")` +1. Compact syntax using the [match operator `:`](elasticsearch://reference/query-languages/esql/functions-operators/operators.md#esql-match-operator): `field:"search terms"` Both are equivalent and can be used interchangeably. The compact syntax is more concise, while the function syntax allows for more configuration options. We'll use the compact syntax in most examples for brevity. -Refer to the [match function](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-match) reference docs for advanced parameters available with the function syntax. +Refer to the [match function](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md#esql-match) reference docs for advanced parameters available with the function syntax. ::: ### Basic full-text query @@ -382,7 +382,7 @@ FROM cooking_blog METADATA _score This tutorial introduced the basics of search and filtering in {{esql}}. Building a real-world search experience requires understanding many more advanced concepts and techniques. Here are some resources once you're ready to dive deeper: - [Search with {{esql}}](esql-for-search.md): Learn about all your options for search use cases with {{esql}}. -- [{{esql}} search functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-functions): Explore the full list of search functions available in {{esql}}. +- [{{esql}} search functions](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md): Explore the full list of search functions available in {{esql}}. - [Semantic search](/solutions/search/semantic-search.md): Understand your various options for semantic search in Elasticsearch. - [The `semantic_text` workflow](/solutions/search/semantic-search.md#_semantic_text_workflow): Learn how to use the `semantic_text` field type for semantic search. This is the recommended approach for most users looking to perform semantic search in {{es}}, because it abstracts away the complexity of setting up inference endpoints and models. diff --git a/solutions/search/full-text.md b/solutions/search/full-text.md index 08892297ba..af925eff38 100644 --- a/solutions/search/full-text.md +++ b/solutions/search/full-text.md @@ -45,7 +45,7 @@ Learn about the core components of full-text search: Learn how to build full-text search queries using {{es}}'s query languages: * [Full-text queries using Query DSL](elasticsearch://reference/query-languages/query-dsl/full-text-queries.md) -* [Full-text search functions in {{esql}}](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-functions) +* [Full-text search functions in {{esql}}](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md) **Advanced topics** diff --git a/solutions/search/full-text/how-full-text-works.md b/solutions/search/full-text/how-full-text-works.md index fce499d5a1..a458e445bf 100644 --- a/solutions/search/full-text/how-full-text-works.md +++ b/solutions/search/full-text/how-full-text-works.md @@ -32,4 +32,4 @@ Refer to [Test an analyzer](../../../manage-data/data-store/text-analysis/test-a Query DSL supports a number of [full-text queries](elasticsearch://reference/query-languages/query-dsl/full-text-queries.md). - As of 8.17, {{esql}} also supports [full-text search](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-functions) functions. + As of 8.17, {{esql}} also supports [full-text search](elasticsearch://reference/query-languages/esql/functions-operators/search-functions.md) functions. diff --git a/solutions/search/hybrid-semantic-text.md b/solutions/search/hybrid-semantic-text.md index 023d504b0c..caa6074550 100644 --- a/solutions/search/hybrid-semantic-text.md +++ b/solutions/search/hybrid-semantic-text.md @@ -228,7 +228,7 @@ POST /_query?format=txt } ``` 1. The `METADATA _score` clause is used to return the score of each document -2. The [match (`:`) operator](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-operators) is used on the `content` field for standard keyword matching +2. The [match (`:`) operator](elasticsearch://reference/query-languages/esql/functions-operators/operators.md#esql-match-operator) is used on the `content` field for standard keyword matching 3. Semantic search using the `match()` function on the `semantic_text` field with a boost of `0.75` 4. Sorts by descending score and limits to 1000 results ::: diff --git a/solutions/search/semantic-search/semantic-search-semantic-text.md b/solutions/search/semantic-search/semantic-search-semantic-text.md index a98c945f18..2c8e35cdf6 100644 --- a/solutions/search/semantic-search/semantic-search-semantic-text.md +++ b/solutions/search/semantic-search/semantic-search-semantic-text.md @@ -127,7 +127,7 @@ GET semantic-embeddings/_search :::{tab-item} ES|QL :sync: esql -The ES|QL approach uses the [match (`:`) operator](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-operators), which automatically detects the `semantic_text` field and performs the search on it. The query uses `METADATA _score` to sort by `_score` in descending order. +The ES|QL approach uses the [match (`:`) operator](elasticsearch://reference/query-languages/esql/functions-operators/operators.md#esql-match-operator), which automatically detects the `semantic_text` field and performs the search on it. The query uses `METADATA _score` to sort by `_score` in descending order. ```console @@ -142,7 +142,7 @@ POST /_query?format=txt } ``` 1. The `METADATA _score` clause is used to return the score of each document -2. The [match (`:`) operator](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-operators) is used on the `content` field for standard keyword matching +2. The [match (`:`) operator](elasticsearch://reference/query-languages/esql/functions-operators/operators.md#esql-match-operator) is used on the `content` field for standard keyword matching 3. Sorts by descending score to display the most relevant results first 4. Limits the results to 1000 documents diff --git a/solutions/security/detect-and-alert/create-detection-rule.md b/solutions/security/detect-and-alert/create-detection-rule.md index 1b5e4c1da9..8716728fda 100644 --- a/solutions/security/detect-and-alert/create-detection-rule.md +++ b/solutions/security/detect-and-alert/create-detection-rule.md @@ -395,7 +395,7 @@ To create an {{esql}} rule: #### Aggregating query [esql-agg-query] -Aggregating queries use [`STATS...BY`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-agg-functions) functions to aggregate source event data. Alerts generated by a rule with an aggregating query only contain the fields that the {{esql}} query returns and any new fields that the query creates. +Aggregating queries use [`STATS...BY`](elasticsearch://reference/query-languages/esql/functions-operators/aggregation-functions.md) functions to aggregate source event data. Alerts generated by a rule with an aggregating query only contain the fields that the {{esql}} query returns and any new fields that the query creates. ::::{note} A *new field* is a field that doesn’t exist in the query’s source index and is instead created when the rule runs. You can access new fields in the details of any alerts that are generated by the rule. For example, if you use the `STATS...BY` function to create a column with aggregated values, the column is created when the rule runs and is added as a new field to any alerts that are generated by the rule. @@ -427,7 +427,7 @@ Rules that use aggregating queries might create duplicate alerts. This can happe Non-aggregating queries don’t use `STATS...BY` functions and don’t aggregate source event data. Alerts generated by a non-aggregating query contain source event fields that the query returns, new fields the query creates, and all other fields in the source event document. ::::{note} -A *new field* is a field that doesn’t exist in the query’s source index and is instead created when the rule runs. You can access new fields in the details of any alerts that are generated by the rule. For example, if you use the [`EVAL`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-eval) command to append new columns with calculated values, the columns are created when the rule runs, and are added as new fields to any alerts generated by the rule. +A *new field* is a field that doesn’t exist in the query’s source index and is instead created when the rule runs. You can access new fields in the details of any alerts that are generated by the rule. For example, if you use the [`EVAL`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-eval) command to append new columns with calculated values, the columns are created when the rule runs, and are added as new fields to any alerts generated by the rule. :::: @@ -456,7 +456,7 @@ FROM logs-* METADATA _id, _index, _version When those metadata fields are provided, unique alert IDs are created for each alert generated by the query. -When developing the query, make sure you don’t [`DROP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-drop) or filter out the `_id`, `_index`, or `_version` metadata fields. +When developing the query, make sure you don’t [`DROP`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-drop) or filter out the `_id`, `_index`, or `_version` metadata fields. Here is an example of a query that fails to deduplicate alerts. It uses the `DROP` command to omit the `_id` property from the results table: @@ -481,11 +481,11 @@ FROM logs-* METADATA _id, _index, _version When writing your query, consider the following: -* The [`LIMIT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-limit) command specifies the maximum number of rows an {{esql}} query returns and the maximum number of alerts created per rule execution. Similarly, a detection rule’s **Max alerts per run** setting specifies the maximum number of alerts it can create every time it runs. +* The [`LIMIT`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-limit) command specifies the maximum number of rows an {{esql}} query returns and the maximum number of alerts created per rule execution. Similarly, a detection rule’s **Max alerts per run** setting specifies the maximum number of alerts it can create every time it runs. If the `LIMIT` value and **Max alerts per run** value are different, the rule uses the lower value to determine the maximum number of alerts the rule generates. -* When writing an aggregating query, use the [`STATS...BY`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-stats-by) command with fields that you want to search and filter for after alerts are created. For example, using the `host.name`, `user.name`, `process.name` fields with the `BY` operator of the `STATS...BY` command returns these fields in alert documents, and allows you to search and filter for them from the Alerts table. +* When writing an aggregating query, use the [`STATS...BY`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-stats-by) command with fields that you want to search and filter for after alerts are created. For example, using the `host.name`, `user.name`, `process.name` fields with the `BY` operator of the `STATS...BY` command returns these fields in alert documents, and allows you to search and filter for them from the Alerts table. * When configuring alert suppression on a non-aggregating query, we recommend sorting results by ascending `@timestamp` order. Doing so ensures that alerts are properly suppressed, especially if the number of alerts generated is higher than the **Max alerts per run** value. diff --git a/solutions/security/investigate/timeline.md b/solutions/security/investigate/timeline.md index bae813fbf5..dc5e16948b 100644 --- a/solutions/security/investigate/timeline.md +++ b/solutions/security/investigate/timeline.md @@ -254,7 +254,7 @@ You can use {{esql}} in Timeline by opening the **{{esql}}** tab. From there, yo * Finally, it keeps the default Timeline fields (`@timestamp`, `message`, `event.category`, `event.action`, `host.name`, `source.ip`, `destination.ip`, and `user.name`) in the output. ::::{tip} - When querying indices that tend to be large (for example, `logs-*`), performance can be impacted by the number of fields returned in the output. To optimize performance, we recommend using the [`KEEP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-keep) command to specify fields that you want returned. For example, add the clause `KEEP @timestamp, user.name` to the end of your query to specify that you only want the `@timestamp` and `user.name` fields returned. + When querying indices that tend to be large (for example, `logs-*`), performance can be impacted by the number of fields returned in the output. To optimize performance, we recommend using the [`KEEP`](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-keep) command to specify fields that you want returned. For example, add the clause `KEEP @timestamp, user.name` to the end of your query to specify that you only want the `@timestamp` and `user.name` fields returned. ::::