-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Add documentation for TS source #134373
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add documentation for TS source #134373
Changes from all commits
6463013
8677ce0
49630b7
3beebc4
4104e1a
c860567
5d23ffe
4ec4daf
2feaffc
d9e9451
fd54d14
7dbd092
54f994b
b299ec4
0e56b47
9b3bb28
e452620
bf342c2
9b557ce
647534c
8afa050
a83f082
5cb5f45
086fece
a376340
020d26b
ef0611f
cd670c1
51b2170
e499726
bd2f557
a9cc818
cd4ed00
8954b44
1d25d54
8545d17
c9a21d4
9e192d9
4d7e593
f32b1ff
ee1df8b
9276b86
b0600a9
aa587e8
1f17af3
71fa400
013a5e0
2b1832c
59541ae
a2706ea
2921961
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -1,12 +1,16 @@ | ||||||||||||
| ```yaml {applies_to} | ||||||||||||
| serverless: ga | ||||||||||||
| stack: ga | ||||||||||||
| serverless: preview | ||||||||||||
| stack: preview 9.2.0 | ||||||||||||
| ``` | ||||||||||||
|
|
||||||||||||
| The `TS` command is similar to the `FROM` source command, | ||||||||||||
| but with two key differences: it targets only [time-series indices](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md) | ||||||||||||
| and enables the use of time-series aggregation functions | ||||||||||||
| with the [STATS](/reference/query-languages/esql/commands/stats-by.md) command. | ||||||||||||
| **Brief description** | ||||||||||||
|
|
||||||||||||
| The `TS` source command is similar to the [`FROM`](/reference/query-languages/esql/commands/from.md) | ||||||||||||
| source command, with the following key differences: | ||||||||||||
|
|
||||||||||||
| - Targets only [time series indices](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md) | ||||||||||||
| - Enables the use of [time series aggregation functions](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md) inside the | ||||||||||||
| [STATS](/reference/query-languages/esql/commands/stats-by.md) command | ||||||||||||
|
|
||||||||||||
| **Syntax** | ||||||||||||
|
|
||||||||||||
|
|
@@ -22,10 +26,92 @@ TS index_pattern [METADATA fields] | |||||||||||
| `fields` | ||||||||||||
| : A comma-separated list of [metadata fields](/reference/query-languages/esql/esql-metadata-fields.md) to retrieve. | ||||||||||||
|
|
||||||||||||
| **Description** | ||||||||||||
|
|
||||||||||||
| The `TS` source command enables time series semantics and adds support for | ||||||||||||
| [time series aggregation functions](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md) to the `STATS` command, such as | ||||||||||||
| [`AVG_OVER_TIME()`](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md#esql-avg_over_time), | ||||||||||||
| or [`RATE`](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md#esql-rate). | ||||||||||||
| These functions are implicitly evaluated per time series, then aggregated by group using a secondary aggregation | ||||||||||||
| function. For example: | ||||||||||||
|
Comment on lines
+35
to
+36
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: maybe improve the wording a bit on this.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll stick with Marci's version. |
||||||||||||
|
|
||||||||||||
| ```esql | ||||||||||||
| TS metrics | ||||||||||||
| | WHERE @timestamp >= now() - 1 hour | ||||||||||||
felixbarny marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||
| | STATS SUM(RATE(search_requests)) BY TBUCKET(1 hour), host | ||||||||||||
| ``` | ||||||||||||
|
|
||||||||||||
| This query calculates the total rate of search requests (tracked by the `search_requests` counter) per host and hour. The `RATE()` | ||||||||||||
| function is applied per time series in hourly buckets. These rates are summed for each | ||||||||||||
| host and hourly bucket (since each host can map to multiple time series). | ||||||||||||
|
|
||||||||||||
| This paradigm—a pair of aggregation functions—is standard for time series | ||||||||||||
| querying. For supported inner (time series) functions per | ||||||||||||
| [metric type](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-metric), refer to | ||||||||||||
| [](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md). These functions also | ||||||||||||
| apply to downsampled data, with the same semantics as for raw data. | ||||||||||||
|
|
||||||||||||
| ::::{note} | ||||||||||||
| If a query is missing an inner (time series) aggregation function, | ||||||||||||
| [`LAST_OVER_TIME()`](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md#esql-last_over_time) | ||||||||||||
| is assumed and used implicitly. For instance, the following two queries are | ||||||||||||
| equivalent, returning the average of the last memory usage values per time series: | ||||||||||||
|
|
||||||||||||
| ```esql | ||||||||||||
| TS metrics | STATS AVG(memory_usage) | ||||||||||||
|
|
||||||||||||
| TS metrics | STATS AVG(LAST_OVER_TIME(memory_usage)) | ||||||||||||
| ``` | ||||||||||||
|
|
||||||||||||
| To calculate the average memory usage across per-time-series averages, use | ||||||||||||
| the following query: | ||||||||||||
|
|
||||||||||||
| ```esql | ||||||||||||
| TS metrics | STATS AVG(AVG_OVER_TIME(memory_usage)) | ||||||||||||
| ``` | ||||||||||||
| :::: | ||||||||||||
|
|
||||||||||||
| Use regular (non-time-series) | ||||||||||||
| [aggregation functions](/reference/query-languages/esql/functions-operators/aggregation-functions.md), | ||||||||||||
| such as `SUM()`, as outer aggregation functions. Using a time series aggregation | ||||||||||||
| in combination with an inner function causes an error. For example, the | ||||||||||||
|
Comment on lines
+76
to
+77
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hm nesting is not very clear? Using them as inner functions can still be called "nesting". |
||||||||||||
| following query is invalid: | ||||||||||||
|
|
||||||||||||
| ```esql | ||||||||||||
| TS metrics | STATS AVG_OVER_TIME(RATE(memory_usage)) | ||||||||||||
| ``` | ||||||||||||
|
|
||||||||||||
| ::::{note} | ||||||||||||
| A [time series](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md) | ||||||||||||
| aggregation function must be wrapped inside a | ||||||||||||
| [regular](/reference/query-languages/esql/functions-operators/aggregation-functions.md) | ||||||||||||
| aggregation function. For instance, the following query is invalid: | ||||||||||||
|
|
||||||||||||
| ```esql | ||||||||||||
| TS metrics | STATS RATE(search_requests) | ||||||||||||
| ``` | ||||||||||||
| :::: | ||||||||||||
|
|
||||||||||||
| **Best practices** | ||||||||||||
|
|
||||||||||||
| - Avoid aggregating multiple metrics in the same query when those metrics have different dimensional cardinalities. | ||||||||||||
| For example, in `STATS max(rate(foo)) + rate(bar))`, if `foo` and `bar` don't share the same dimension values, the rate | ||||||||||||
| for one metric will be null for some dimension combinations. Because the + operator returns null when either input | ||||||||||||
| is null, the entire result becomes null for those dimensions. Additionally, queries that aggregate a single metric | ||||||||||||
| can filter out null values more efficiently. | ||||||||||||
kkrik-es marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||
| - Use the `TS` command for aggregations on time series data, rather than `FROM`. The `FROM` command is still available | ||||||||||||
| (for example, for listing document contents), but it's not optimized for procesing time series data and may produce | ||||||||||||
| unexpected results. | ||||||||||||
| - The `TS` command can't be combined with certain operations (such as | ||||||||||||
| [`FORK`](/reference/query-languages/esql/commands/fork.md)) before the `STATS` command is applied. Once `STATS` is | ||||||||||||
| applied, you can process the tabular output with any applicable ES|QL operations. | ||||||||||||
| - Add a time range filter on `@timestamp` to limit the data volume scanned and improve query performance. | ||||||||||||
|
|
||||||||||||
| **Examples** | ||||||||||||
|
|
||||||||||||
| ```esql | ||||||||||||
| TS metrics | ||||||||||||
| | STATS sum(last_over_time(memory_usage)) | ||||||||||||
| | WHERE @timestamp >= now() - 1 day | ||||||||||||
| | STATS SUM(AVG_OVER_TIME(memory_usage)) BY host, TBUCKET(1 hour) | ||||||||||||
| ``` | ||||||||||||
|
|
||||||||||||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Uh oh!
There was an error while loading. Please reload this page.