You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
-`create_doc` and `create_index` for creating or converting a TSDS
25
24
-`manage` to [roll over](#convert-existing-data-stream-to-tsds) a TSDS
26
25
27
26
## Set up a TSDS
28
27
29
28
:::::{stepper}
30
-
::::{step} Create an index lifecycle policy (optional)
31
-
:anchor: tsds-ilm-policy
32
-
33
-
```{applies_to}
34
-
stack: ga
35
-
serverless: unavailable
36
-
```
37
-
In most cases, you can use a data stream lifecycle to manage your time series data stream. If you're using [data tiers](/manage-data/lifecycle/data-tiers.md) in {{stack}}, you can use index lifecycle management (ILM).
38
-
39
-
:::{dropdown} Create an ILM policy
40
-
41
-
If you're using {{stack}}, {{ilm-init}} can help you manage a time series data stream's backing indices. {{ilm-init}} requires an index lifecycle policy.
42
-
43
-
For best results, specify a `max_age` criteria for the `rollover` action in the policy. This ensures the [`@timestamp` ranges](/manage-data/data-store/data-streams/time-bound-tsds.md) for the backing indices are consistent. For example, setting a `max_age` of `1d` for the `rollover` action ensures your backing indices consistently contain one day's worth of data.
44
-
45
-
**Example:**
46
-
47
-
```console
48
-
PUT _ilm/policy/my-weather-sensor-lifecycle-policy
49
-
{
50
-
"policy": {
51
-
"phases": {
52
-
"hot": {
53
-
"actions": {
54
-
"rollover": {
55
-
"max_age": "1d",
56
-
"max_primary_shard_size": "50gb"
57
-
}
58
-
}
59
-
},
60
-
"warm": {
61
-
"min_age": "30d",
62
-
"actions": {
63
-
"shrink": {
64
-
"number_of_shards": 1
65
-
},
66
-
"forcemerge": {
67
-
"max_num_segments": 1
68
-
}
69
-
}
70
-
},
71
-
"cold": {
72
-
"min_age": "60d",
73
-
"actions": {
74
-
"searchable_snapshot": {
75
-
"snapshot_repository": "found-snapshots"
76
-
}
77
-
}
78
-
},
79
-
"frozen": {
80
-
"min_age": "90d",
81
-
"actions": {
82
-
"searchable_snapshot": {
83
-
"snapshot_repository": "found-snapshots"
84
-
}
85
-
}
86
-
},
87
-
"delete": {
88
-
"min_age": "735d",
89
-
"actions": {
90
-
"delete": {}
91
-
}
92
-
}
93
-
}
94
-
}
95
-
}
96
-
```
97
-
:::
98
-
99
-
::::
100
-
101
29
::::{step} Create an index template
102
30
:anchor: create-tsds-index-template
103
31
@@ -106,9 +34,10 @@ The structure of a time series data stream is defined by an index template. Crea
106
34
-**Index patterns:** One or more wildcard patterns matching the name of your TSDS, such as `weather-sernsors-*`. For best results, use the [data stream naming scheme](/reference/fleet/data-streams.md#data-streams-naming-scheme).
107
35
-**Data stream object:** The template must include `"data_stream": {}`.
108
36
-**Time series mode:** Set `index.mode: time_series`.
109
-
-**Field mappings:** Define at least one `keyword` dimension field and typically one or more metric fields:
110
-
- To define a dimension, set `time_series_dimension` to `true`. Dimension fields like `counter` only increase over time. For more details, refer to [Dimensions](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension).
111
-
- To define a metric, use the `time_series_metric` mapping parameter. Metric fields like `gauge` can increase or decrease over time. For more details, refer to [Metrics](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-metric).
37
+
-**Field mappings:** Define at least one dimension field and typically one or more metric fields:
38
+
- To define a dimension, set `time_series_dimension` to `true`. For more details, refer to [Dimensions](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension).
39
+
- To define dimensions dynamically, you can use a pass-through object. For details, refer to [Defining sub-fields as time series dimensions](elasticsearch://reference/elasticsearch/mapping-reference/passthrough.md#passthrough-dimensions).
40
+
- To define a metric, use the `time_series_metric` mapping parameter. For more details, refer to [Metrics](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-metric).
112
41
- (Optional) Define a `date` or `date_nanos` mapping for the `@timestamp` field. If you don't specify a mapping, {{es}} maps `@timestamp` as a `date` field with default options.
113
42
* (Optional) Other index settings, such as [`index.number_of_replicas`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas), for the data stream's backing indices.
114
43
- A priority higher than `200`, to avoid [collisions](/manage-data/data-store/templates.md#avoid-index-pattern-collisions) with built-in templates.
@@ -173,7 +102,7 @@ If you're using component templates with a time series data stream, check the fo
173
102
After creating the index template, you can create a time series data stream by [indexing a document](use-data-stream.md#add-documents-to-a-data-stream). The TSDS is created automatically when you index the first document, as long as the index name matches the index template pattern. You can use a bulk API request or a POST request.
174
103
175
104
:::{important}
176
-
To test the following `_bulk` example, update the timestamps to within three hours of your current time. Data added to a TSDS must fit the [accepted time range](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-accepted-time-range).
105
+
To test the following `_bulk` example, update the timestamps to within two hours of your current time. Data added to a TSDS must fit the [accepted time range](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-accepted-time-range).
177
106
:::
178
107
179
108
```console
@@ -231,7 +160,7 @@ GET metrics-prod/_search
231
160
232
161
You can convert an existing regular data stream to a TSDS. Follow these steps:
233
162
234
-
1. Update your existing index template to include time series settings. Also update your index lifecycle policy (if any) and component templates (if any).
163
+
1. Update your existing index template and component templates (if any) to include time series settings.
235
164
2. Use the [rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover) to manually roll over the existing data stream's write index, to apply the changes you made in step 1:
236
165
237
166
```console
@@ -246,9 +175,11 @@ After the rollover, new backing indices will have time series functionality. Exi
246
175
247
176
To control access to a TSDS, use [index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices). Privileges set on a TSDS also apply to the backing indices.
248
177
249
-
For an example, refer to [Data stream privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges).
178
+
For an example, refer to [Data stream privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges).
179
+
180
+
### Set up lifecycle management
250
181
251
-
% TODO Common patterns for time series data streams
182
+
In most cases, you can use a [data stream lifecycle](/manage-data/lifecycle/data-stream.md) to manage your time series data stream. If you're using [data tiers](/manage-data/lifecycle/data-tiers.md) in {{stack}}, you can use [index lifecycle management](/manage-data/lifecycle/index-lifecycle-management.md).
-[Shrink](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md), which might then revert the read-only status at the end of the action
54
54
55
55
{{ilm-cap}} will **not** proceed with executing these actions until [`index.time_series.end_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has passed.
Copy file name to clipboardExpand all lines: manage-data/data-store/data-streams/time-series-data-stream-tsds.md
+15-23Lines changed: 15 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,7 +19,7 @@ Before setting up a time series data stream, make sure you're familiar with gene
19
19
20
20
## When to use a time series data stream [when-to-use-tsds]
21
21
22
-
_Metrics_ consist of data point–timestamp pairs, identified by [dimension fields]() that can be used in aggregation queries. Both a regular data stream and a time series data stream can store metrics data.
22
+
_Metrics_ consist of data point–timestamp pairs, identified by [dimension fields](#time-series-dimension) that can be used in aggregation queries. Both a regular data stream and a time series data stream can store metrics data.
23
23
24
24
Choose a time series data stream if you typically add metrics data to {{es}} in near real-time and in `@timestamp` order. For other timestamped data, such as logs or traces, use a [logs data stream](logs-data-stream.md) or [regular data stream](/manage-data/data-store/data-streams.md).
25
25
@@ -38,11 +38,9 @@ A time series is a sequence of observations for a specific entity. Together, the
38
38
:title: Time series of weather sensor readings plotted as a graph
39
39
:::
40
40
41
-
In a TSDS, each {{es}} document represents an observation, or data point, in a specific time series. Although a TSDS can contain multiple time series, a document can belong to only one time series. A single time series can't span multiple data streams.
42
-
43
41
### Time series fields
44
42
45
-
Compared to a regular data stream, a TSDS uses some additional fields specific to time series: dimension fields (required) and metric fields (optional but usually defined), plus an internal `_tsid` metadata field.
43
+
Compared to a regular data stream, a TSDS uses some additional fields specific to time series: dimension fields and metric fields, plus an internal `_tsid` metadata field.
46
44
47
45
#### Dimensions [time-series-dimension]
48
46
@@ -63,62 +61,56 @@ To mark a field as a dimension, set the Boolean `time_series_dimension` mapping
To work with a flattened field, use the `time_series_dimensions` parameter to configure an array of fields as dimensions. For details, refer to [`flattened`](elasticsearch://reference/elasticsearch/mapping-reference/flattened.md#flattened-params).
68
65
69
66
You can also simplify dimension definitions by using [pass-through](elasticsearch://reference/elasticsearch/mapping-reference/passthrough.md#passthrough-dimensions) fields.
70
-
:::
71
67
72
68
#### Metrics [time-series-metric]
73
69
74
-
Metrics are numeric measurements that change over time. Documents in a TSDS contain one or more metric fields.
70
+
Metrics are numeric measurements that change over time. Documents in a TSDS typically contain one or more metric fields.
75
71
76
72
:::{tip}
77
73
Metrics are expected to change (even if rarely or slowly), while dimensions generally remain constant.
78
74
:::
79
75
80
76
To mark a field as a metric, use the `time_series_metric` mapping parameter. This parameter ensures data is stored in an optimal way for time series analysis. The following field types support the `time_series_metric` parameter:
* All [numeric field types](elasticsearch://reference/elasticsearch/mapping-reference/number.md)
79
+
*[`aggregate_metric_double`](elasticsearch://reference/elasticsearch/mapping-reference/aggregate-metric-double.md), for internal use during downsampling (rarely user-populated)
84
80
85
81
The valid values for `time_series_metric` are `counter` and `gauge`:
86
82
87
83
`counter`
88
-
: A cumulative metric that only monotonically increases or resets to `0` (zero). For example, a count of errors or completed tasks.
84
+
: A cumulative metric that only monotonically increases or resets to `0` (zero). For example, a count of errors or completed tasks that resets when a serving process restarts.
89
85
90
86
`gauge`
91
87
: A metric that represents a single numeric that can arbitrarily increase or decrease. For example, a temperature or available disk space.
92
88
93
89
#### `_tsid` metadata field [tsid]
94
90
95
-
The `_tsid` is an automatically generated object containing the document’s dimensions. It's intended for internal {{es}} use, so in most cases you won't need to work with it.
96
-
97
-
- You **can't** query or update the internal `_tsid` field.
98
-
- You **can** use the `_tsid` in aggregations.
99
-
- To retrieve the value of `_tsid`, use the fields parameter in a search. The `_tsid` is not included in get document responses.
100
-
- The format of the `_tsid` field is subject to change.
91
+
The `_tsid` is an automatically generated object containing the document’s dimensions. It's intended for internal {{es}} use, so in most cases you won't need to work with it. The format of the `_tsid` field is subject to change.
101
92
102
93
### Differences from a regular data stream [differences-from-regular-data-stream]
103
94
104
95
A time series data stream works like a regular data stream, with some key differences:
105
96
106
97
***Time series index mode:** The matching index template for a TSDS must include a `data_stream` object with `index.mode` set to `time_series`. This option enables most TSDS-related functionality.
107
-
***Required fields:** In a TSDS, each document contains:
98
+
***Fields:** In a TSDS, each document contains:
108
99
* A `@timestamp` field
109
100
* One or more [dimension fields](#time-series-dimension), set with `time_series_dimension: true`
110
-
* One or more [metric fields](#time-series-metric) (not strictly required, but typical for a TSDS)
111
-
***Document IDs:** Time series documents use two IDs:
112
-
* An internal [`_tsid`](#tsid) metadata field, generated by {{es}} for each document in a TSDS and used for sorting and compression
113
-
* The document `_id`, a generated hash of the document's dimensions and `@timestamp` (custom `_id` values are not supported)
101
+
* One or more [metric fields](#time-series-metric)
102
+
* An auto-generated document `_id` (custom `_id` values are not supported)
114
103
***Backing indices:** A TSDS uses [time-bound indices](/manage-data/data-store/data-streams/time-bound-tsds.md) to store data from the same time period in the same backing index.
115
-
***Dimension-based routing:** The matching index template for a TSDS must contain the `index.routing_path` index setting, which specifies dimensions for routing documents to shards.
104
+
***Dimension-based routing:** The routing logic uses dimension fields to map data to shards, improving storage efficiency and query performance.
116
105
***Sorting:** A TSDS uses internal [index sorting](elasticsearch://reference/elasticsearch/index-settings/sorting.md) to order shard segments by `_tsid` and `@timestamp`, for better compression. Time series data streams do not use `index.sort.*` settings.
117
-
***Synthetic source:** A TSDS uses [synthetic `_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source), which has some [restrictions](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-restrictions) and [modifications](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications).
118
106
119
107
## Query time series data
108
+
```{applies_to}
109
+
stack: preview
110
+
serverless: preview
111
+
```
120
112
121
-
You can use the {{esql}} [`TS` command](elasticsearch://reference/query-languages/esql/commands/ts.md) to query time series data streams. The `TS` command is optimized for time series data. It also enables the use of aggregation functions that efficiently process metrics per time series, before aggregating results.
113
+
You can use the {{esql}} [`TS` command](elasticsearch://reference/query-languages/esql/commands/ts.md)(in technical preview) to query time series data streams. The `TS` command is optimized for time series data. It also enables the use of aggregation functions that efficiently process metrics per time series, before aggregating results.
0 commit comments