Skip to content

Commit b659c3a

Browse files
kkrik-esmarciwleemthompofelixbarny
authored
Add documentation for TS source (#134373)
* Add documentation for TS source * add toc ref * update nav * update * Apply suggestions from code review Co-authored-by: Marci W <[email protected]> * Update ts.md * Apply suggestions from code review Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Felix Barnsteiner <[email protected]> * update * add links to time-series functions * restore brief description * fix formatting * remove links * Apply suggestions from code review Co-authored-by: Felix Barnsteiner <[email protected]> * Update ts.md * Update ts.md * Update ts.md * update * mark over time functions as preview * add time-series aggs * list time-series aggs * add missing * add toc * remove snapshot notes * add title * sum_over_time example * fix ts agg refs * add links to TS doc * fix links to TS doc * fix links to TS doc * remove note * Apply suggestions from code review Co-authored-by: Marci W <[email protected]> * Update ts.md * Apply suggestions from code review Co-authored-by: Liam Thompson <[email protected]> * Apply suggestions from code review Co-authored-by: Liam Thompson <[email protected]> * update function description * Update docs/reference/query-languages/esql/_snippets/commands/layout/ts.md Co-authored-by: Liam Thompson <[email protected]> * Apply suggestions from code review Co-authored-by: Felix Barnsteiner <[email protected]> * fix from review * Update ts.md --------- Co-authored-by: Marci W <[email protected]> Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Felix Barnsteiner <[email protected]>
1 parent 0997c84 commit b659c3a

File tree

78 files changed

+269
-196
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

78 files changed

+269
-196
lines changed

docs/reference/query-languages/esql/_snippets/commands/layout/ts.md

Lines changed: 93 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,16 @@
11
```yaml {applies_to}
2-
serverless: ga
3-
stack: ga
2+
serverless: preview
3+
stack: preview 9.2.0
44
```
55
6-
The `TS` command is similar to the `FROM` source command,
7-
but with two key differences: it targets only [time-series indices](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md)
8-
and enables the use of time-series aggregation functions
9-
with the [STATS](/reference/query-languages/esql/commands/stats-by.md) command.
6+
**Brief description**
7+
8+
The `TS` source command is similar to the [`FROM`](/reference/query-languages/esql/commands/from.md)
9+
source command, with the following key differences:
10+
11+
- Targets only [time series indices](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md)
12+
- Enables the use of [time series aggregation functions](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md) inside the
13+
[STATS](/reference/query-languages/esql/commands/stats-by.md) command
1014

1115
**Syntax**
1216

@@ -22,10 +26,92 @@ TS index_pattern [METADATA fields]
2226
`fields`
2327
: A comma-separated list of [metadata fields](/reference/query-languages/esql/esql-metadata-fields.md) to retrieve.
2428

29+
**Description**
30+
31+
The `TS` source command enables time series semantics and adds support for
32+
[time series aggregation functions](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md) to the `STATS` command, such as
33+
[`AVG_OVER_TIME()`](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md#esql-avg_over_time),
34+
or [`RATE`](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md#esql-rate).
35+
These functions are implicitly evaluated per time series, then aggregated by group using a secondary aggregation
36+
function. For example:
37+
38+
```esql
39+
TS metrics
40+
| WHERE @timestamp >= now() - 1 hour
41+
| STATS SUM(RATE(search_requests)) BY TBUCKET(1 hour), host
42+
```
43+
44+
This query calculates the total rate of search requests (tracked by the `search_requests` counter) per host and hour. The `RATE()`
45+
function is applied per time series in hourly buckets. These rates are summed for each
46+
host and hourly bucket (since each host can map to multiple time series).
47+
48+
This paradigm—a pair of aggregation functions—is standard for time series
49+
querying. For supported inner (time series) functions per
50+
[metric type](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-metric), refer to
51+
[](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md). These functions also
52+
apply to downsampled data, with the same semantics as for raw data.
53+
54+
::::{note}
55+
If a query is missing an inner (time series) aggregation function,
56+
[`LAST_OVER_TIME()`](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md#esql-last_over_time)
57+
is assumed and used implicitly. For instance, the following two queries are
58+
equivalent, returning the average of the last memory usage values per time series:
59+
60+
```esql
61+
TS metrics | STATS AVG(memory_usage)
62+
63+
TS metrics | STATS AVG(LAST_OVER_TIME(memory_usage))
64+
```
65+
66+
To calculate the average memory usage across per-time-series averages, use
67+
the following query:
68+
69+
```esql
70+
TS metrics | STATS AVG(AVG_OVER_TIME(memory_usage))
71+
```
72+
::::
73+
74+
Use regular (non-time-series)
75+
[aggregation functions](/reference/query-languages/esql/functions-operators/aggregation-functions.md),
76+
such as `SUM()`, as outer aggregation functions. Using a time series aggregation
77+
in combination with an inner function causes an error. For example, the
78+
following query is invalid:
79+
80+
```esql
81+
TS metrics | STATS AVG_OVER_TIME(RATE(memory_usage))
82+
```
83+
84+
::::{note}
85+
A [time series](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md)
86+
aggregation function must be wrapped inside a
87+
[regular](/reference/query-languages/esql/functions-operators/aggregation-functions.md)
88+
aggregation function. For instance, the following query is invalid:
89+
90+
```esql
91+
TS metrics | STATS RATE(search_requests)
92+
```
93+
::::
94+
95+
**Best practices**
96+
97+
- Avoid aggregating multiple metrics in the same query when those metrics have different dimensional cardinalities.
98+
For example, in `STATS max(rate(foo)) + rate(bar))`, if `foo` and `bar` don't share the same dimension values, the rate
99+
for one metric will be null for some dimension combinations. Because the + operator returns null when either input
100+
is null, the entire result becomes null for those dimensions. Additionally, queries that aggregate a single metric
101+
can filter out null values more efficiently.
102+
- Use the `TS` command for aggregations on time series data, rather than `FROM`. The `FROM` command is still available
103+
(for example, for listing document contents), but it's not optimized for procesing time series data and may produce
104+
unexpected results.
105+
- The `TS` command can't be combined with certain operations (such as
106+
[`FORK`](/reference/query-languages/esql/commands/fork.md)) before the `STATS` command is applied. Once `STATS` is
107+
applied, you can process the tabular output with any applicable ES|QL operations.
108+
- Add a time range filter on `@timestamp` to limit the data volume scanned and improve query performance.
109+
25110
**Examples**
26111

27112
```esql
28113
TS metrics
29-
| STATS sum(last_over_time(memory_usage))
114+
| WHERE @timestamp >= now() - 1 day
115+
| STATS SUM(AVG_OVER_TIME(memory_usage)) BY host, TBUCKET(1 hour)
30116
```
31117

docs/reference/query-languages/esql/_snippets/functions/description/absent_over_time.md

Lines changed: 1 addition & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/query-languages/esql/_snippets/functions/description/avg_over_time.md

Lines changed: 1 addition & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/query-languages/esql/_snippets/functions/description/count_distinct_over_time.md

Lines changed: 1 addition & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/query-languages/esql/_snippets/functions/description/count_over_time.md

Lines changed: 1 addition & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/query-languages/esql/_snippets/functions/description/first.md

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/query-languages/esql/_snippets/functions/description/first_over_time.md

Lines changed: 1 addition & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/query-languages/esql/_snippets/functions/description/last.md

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/query-languages/esql/_snippets/functions/description/last_over_time.md

Lines changed: 1 addition & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/query-languages/esql/_snippets/functions/description/max_over_time.md

Lines changed: 1 addition & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)