Skip to content

Commit 947e8d5

Browse files
committed
Address comments and suggestions from reviewers
1 parent 7141fff commit 947e8d5

File tree

3 files changed

+27
-104
lines changed

3 files changed

+27
-104
lines changed

manage-data/data-store/data-streams/set-up-tsds.md

Lines changed: 10 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -19,85 +19,13 @@ This page shows you how to set up a [time series data stream](/manage-data/data-
1919
- Make sure you have the following permissions:
2020
- [Cluster privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-cluster)
2121
- `manage_index_templates` for creating a template to base the TSDS on
22-
- `manage_ilm` (Stack only) if you're using [index lifecycle management](#tsds-ilm-policy)
2322
- [Index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices)
2423
- `create_doc` and `create_index` for creating or converting a TSDS
2524
- `manage` to [roll over](#convert-existing-data-stream-to-tsds) a TSDS
2625

2726
## Set up a TSDS
2827

2928
:::::{stepper}
30-
::::{step} Create an index lifecycle policy (optional)
31-
:anchor: tsds-ilm-policy
32-
33-
```{applies_to}
34-
stack: ga
35-
serverless: unavailable
36-
```
37-
In most cases, you can use a data stream lifecycle to manage your time series data stream. If you're using [data tiers](/manage-data/lifecycle/data-tiers.md) in {{stack}}, you can use index lifecycle management (ILM).
38-
39-
:::{dropdown} Create an ILM policy
40-
41-
If you're using {{stack}}, {{ilm-init}} can help you manage a time series data stream's backing indices. {{ilm-init}} requires an index lifecycle policy.
42-
43-
For best results, specify a `max_age` criteria for the `rollover` action in the policy. This ensures the [`@timestamp` ranges](/manage-data/data-store/data-streams/time-bound-tsds.md) for the backing indices are consistent. For example, setting a `max_age` of `1d` for the `rollover` action ensures your backing indices consistently contain one day's worth of data.
44-
45-
**Example:**
46-
47-
```console
48-
PUT _ilm/policy/my-weather-sensor-lifecycle-policy
49-
{
50-
"policy": {
51-
"phases": {
52-
"hot": {
53-
"actions": {
54-
"rollover": {
55-
"max_age": "1d",
56-
"max_primary_shard_size": "50gb"
57-
}
58-
}
59-
},
60-
"warm": {
61-
"min_age": "30d",
62-
"actions": {
63-
"shrink": {
64-
"number_of_shards": 1
65-
},
66-
"forcemerge": {
67-
"max_num_segments": 1
68-
}
69-
}
70-
},
71-
"cold": {
72-
"min_age": "60d",
73-
"actions": {
74-
"searchable_snapshot": {
75-
"snapshot_repository": "found-snapshots"
76-
}
77-
}
78-
},
79-
"frozen": {
80-
"min_age": "90d",
81-
"actions": {
82-
"searchable_snapshot": {
83-
"snapshot_repository": "found-snapshots"
84-
}
85-
}
86-
},
87-
"delete": {
88-
"min_age": "735d",
89-
"actions": {
90-
"delete": {}
91-
}
92-
}
93-
}
94-
}
95-
}
96-
```
97-
:::
98-
99-
::::
100-
10129
::::{step} Create an index template
10230
:anchor: create-tsds-index-template
10331

@@ -106,9 +34,10 @@ The structure of a time series data stream is defined by an index template. Crea
10634
- **Index patterns:** One or more wildcard patterns matching the name of your TSDS, such as `weather-sernsors-*`. For best results, use the [data stream naming scheme](/reference/fleet/data-streams.md#data-streams-naming-scheme).
10735
- **Data stream object:** The template must include `"data_stream": {}`.
10836
- **Time series mode:** Set `index.mode: time_series`.
109-
- **Field mappings:** Define at least one `keyword` dimension field and typically one or more metric fields:
110-
- To define a dimension, set `time_series_dimension` to `true`. Dimension fields like `counter` only increase over time. For more details, refer to [Dimensions](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension).
111-
- To define a metric, use the `time_series_metric` mapping parameter. Metric fields like `gauge` can increase or decrease over time. For more details, refer to [Metrics](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-metric).
37+
- **Field mappings:** Define at least one dimension field and typically one or more metric fields:
38+
- To define a dimension, set `time_series_dimension` to `true`. For more details, refer to [Dimensions](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension).
39+
- To define dimensions dynamically, you can use a pass-through object. For details, refer to [Defining sub-fields as time series dimensions](elasticsearch://reference/elasticsearch/mapping-reference/passthrough.md#passthrough-dimensions).
40+
- To define a metric, use the `time_series_metric` mapping parameter. For more details, refer to [Metrics](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-metric).
11241
- (Optional) Define a `date` or `date_nanos` mapping for the `@timestamp` field. If you don't specify a mapping, {{es}} maps `@timestamp` as a `date` field with default options.
11342
* (Optional) Other index settings, such as [`index.number_of_replicas`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas), for the data stream's backing indices.
11443
- A priority higher than `200`, to avoid [collisions](/manage-data/data-store/templates.md#avoid-index-pattern-collisions) with built-in templates.
@@ -173,7 +102,7 @@ If you're using component templates with a time series data stream, check the fo
173102
After creating the index template, you can create a time series data stream by [indexing a document](use-data-stream.md#add-documents-to-a-data-stream). The TSDS is created automatically when you index the first document, as long as the index name matches the index template pattern. You can use a bulk API request or a POST request.
174103

175104
:::{important}
176-
To test the following `_bulk` example, update the timestamps to within three hours of your current time. Data added to a TSDS must fit the [accepted time range](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-accepted-time-range).
105+
To test the following `_bulk` example, update the timestamps to within two hours of your current time. Data added to a TSDS must fit the [accepted time range](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-accepted-time-range).
177106
:::
178107

179108
```console
@@ -231,7 +160,7 @@ GET metrics-prod/_search
231160

232161
You can convert an existing regular data stream to a TSDS. Follow these steps:
233162

234-
1. Update your existing index template to include time series settings. Also update your index lifecycle policy (if any) and component templates (if any).
163+
1. Update your existing index template and component templates (if any) to include time series settings.
235164
2. Use the [rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover) to manually roll over the existing data stream's write index, to apply the changes you made in step 1:
236165

237166
```console
@@ -246,9 +175,11 @@ After the rollover, new backing indices will have time series functionality. Exi
246175

247176
To control access to a TSDS, use [index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices). Privileges set on a TSDS also apply to the backing indices.
248177

249-
For an example, refer to [Data stream privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges).
178+
For an example, refer to [Data stream privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges).
179+
180+
### Set up lifecycle management
250181

251-
% TODO Common patterns for time series data streams
182+
In most cases, you can use a [data stream lifecycle](/manage-data/lifecycle/data-stream.md) to manage your time series data stream. If you're using [data tiers](/manage-data/lifecycle/data-tiers.md) in {{stack}}, you can use [index lifecycle management](/manage-data/lifecycle/index-lifecycle-management.md).
252183

253184
## Next steps [set-up-tsds-whats-next]
254185

manage-data/data-store/data-streams/time-bound-tsds.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,13 +44,13 @@ GET _data_stream/my-tsds
4444
```
4545

4646
::::{tip}
47-
These {{ilm-init}} actions mark the source index as read-only or prevent writes for performance reasons:
47+
The following actions affect the writable time range of a TSDS, either because they make a backing index read-only or remove it:
4848
- [Delete](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-delete.md)
4949
- [Downsample](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md)
5050
- [Force merge](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md)
5151
- [Read only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md)
5252
- [Searchable snapshot](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md)
53-
- [Shrink](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md)
53+
- [Shrink](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md), which might then revert the read-only status at the end of the action
5454

5555
{{ilm-cap}} will **not** proceed with executing these actions until [`index.time_series.end_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has passed.
5656
::::

manage-data/data-store/data-streams/time-series-data-stream-tsds.md

Lines changed: 15 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ Before setting up a time series data stream, make sure you're familiar with gene
1919

2020
## When to use a time series data stream [when-to-use-tsds]
2121

22-
_Metrics_ consist of data point–timestamp pairs, identified by [dimension fields]() that can be used in aggregation queries. Both a regular data stream and a time series data stream can store metrics data.
22+
_Metrics_ consist of data point–timestamp pairs, identified by [dimension fields](#time-series-dimension) that can be used in aggregation queries. Both a regular data stream and a time series data stream can store metrics data.
2323

2424
Choose a time series data stream if you typically add metrics data to {{es}} in near real-time and in `@timestamp` order. For other timestamped data, such as logs or traces, use a [logs data stream](logs-data-stream.md) or [regular data stream](/manage-data/data-store/data-streams.md).
2525

@@ -38,11 +38,9 @@ A time series is a sequence of observations for a specific entity. Together, the
3838
:title: Time series of weather sensor readings plotted as a graph
3939
:::
4040

41-
In a TSDS, each {{es}} document represents an observation, or data point, in a specific time series. Although a TSDS can contain multiple time series, a document can belong to only one time series. A single time series can't span multiple data streams.
42-
4341
### Time series fields
4442

45-
Compared to a regular data stream, a TSDS uses some additional fields specific to time series: dimension fields (required) and metric fields (optional but usually defined), plus an internal `_tsid` metadata field.
43+
Compared to a regular data stream, a TSDS uses some additional fields specific to time series: dimension fields and metric fields, plus an internal `_tsid` metadata field.
4644

4745
#### Dimensions [time-series-dimension]
4846

@@ -63,62 +61,56 @@ To mark a field as a dimension, set the Boolean `time_series_dimension` mapping
6361
* [`unsigned_long`](elasticsearch://reference/elasticsearch/mapping-reference/number.md)
6462
* [`boolean`](elasticsearch://reference/elasticsearch/mapping-reference/boolean.md)
6563

66-
:::{dropdown} Advanced field types
6764
To work with a flattened field, use the `time_series_dimensions` parameter to configure an array of fields as dimensions. For details, refer to [`flattened`](elasticsearch://reference/elasticsearch/mapping-reference/flattened.md#flattened-params).
6865

6966
You can also simplify dimension definitions by using [pass-through](elasticsearch://reference/elasticsearch/mapping-reference/passthrough.md#passthrough-dimensions) fields.
70-
:::
7167

7268
#### Metrics [time-series-metric]
7369

74-
Metrics are numeric measurements that change over time. Documents in a TSDS contain one or more metric fields.
70+
Metrics are numeric measurements that change over time. Documents in a TSDS typically contain one or more metric fields.
7571

7672
:::{tip}
7773
Metrics are expected to change (even if rarely or slowly), while dimensions generally remain constant.
7874
:::
7975

8076
To mark a field as a metric, use the `time_series_metric` mapping parameter. This parameter ensures data is stored in an optimal way for time series analysis. The following field types support the `time_series_metric` parameter:
8177

82-
* [`aggregate_metric_double`](elasticsearch://reference/elasticsearch/mapping-reference/aggregate-metric-double.md)
8378
* All [numeric field types](elasticsearch://reference/elasticsearch/mapping-reference/number.md)
79+
* [`aggregate_metric_double`](elasticsearch://reference/elasticsearch/mapping-reference/aggregate-metric-double.md), for internal use during downsampling (rarely user-populated)
8480

8581
The valid values for `time_series_metric` are `counter` and `gauge`:
8682

8783
`counter`
88-
: A cumulative metric that only monotonically increases or resets to `0` (zero). For example, a count of errors or completed tasks.
84+
: A cumulative metric that only monotonically increases or resets to `0` (zero). For example, a count of errors or completed tasks that resets when a serving process restarts.
8985

9086
`gauge`
9187
: A metric that represents a single numeric that can arbitrarily increase or decrease. For example, a temperature or available disk space.
9288

9389
#### `_tsid` metadata field [tsid]
9490

95-
The `_tsid` is an automatically generated object containing the document’s dimensions. It's intended for internal {{es}} use, so in most cases you won't need to work with it.
96-
97-
- You **can't** query or update the internal `_tsid` field.
98-
- You **can** use the `_tsid` in aggregations.
99-
- To retrieve the value of `_tsid`, use the fields parameter in a search. The `_tsid` is not included in get document responses.
100-
- The format of the `_tsid` field is subject to change.
91+
The `_tsid` is an automatically generated object containing the document’s dimensions. It's intended for internal {{es}} use, so in most cases you won't need to work with it. The format of the `_tsid` field is subject to change.
10192

10293
### Differences from a regular data stream [differences-from-regular-data-stream]
10394

10495
A time series data stream works like a regular data stream, with some key differences:
10596

10697
* **Time series index mode:** The matching index template for a TSDS must include a `data_stream` object with `index.mode` set to `time_series`. This option enables most TSDS-related functionality.
107-
* **Required fields:** In a TSDS, each document contains:
98+
* **Fields:** In a TSDS, each document contains:
10899
* A `@timestamp` field
109100
* One or more [dimension fields](#time-series-dimension), set with `time_series_dimension: true`
110-
* One or more [metric fields](#time-series-metric) (not strictly required, but typical for a TSDS)
111-
* **Document IDs:** Time series documents use two IDs:
112-
* An internal [`_tsid`](#tsid) metadata field, generated by {{es}} for each document in a TSDS and used for sorting and compression
113-
* The document `_id`, a generated hash of the document's dimensions and `@timestamp` (custom `_id` values are not supported)
101+
* One or more [metric fields](#time-series-metric)
102+
* An auto-generated document `_id` (custom `_id` values are not supported)
114103
* **Backing indices:** A TSDS uses [time-bound indices](/manage-data/data-store/data-streams/time-bound-tsds.md) to store data from the same time period in the same backing index.
115-
* **Dimension-based routing:** The matching index template for a TSDS must contain the `index.routing_path` index setting, which specifies dimensions for routing documents to shards.
104+
* **Dimension-based routing:** The routing logic uses dimension fields to map data to shards, improving storage efficiency and query performance.
116105
* **Sorting:** A TSDS uses internal [index sorting](elasticsearch://reference/elasticsearch/index-settings/sorting.md) to order shard segments by `_tsid` and `@timestamp`, for better compression. Time series data streams do not use `index.sort.*` settings.
117-
* **Synthetic source:** A TSDS uses [synthetic `_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source), which has some [restrictions](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-restrictions) and [modifications](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications).
118106

119107
## Query time series data
108+
```{applies_to}
109+
stack: preview
110+
serverless: preview
111+
```
120112

121-
You can use the {{esql}} [`TS` command](elasticsearch://reference/query-languages/esql/commands/ts.md) to query time series data streams. The `TS` command is optimized for time series data. It also enables the use of aggregation functions that efficiently process metrics per time series, before aggregating results.
113+
You can use the {{esql}} [`TS` command](elasticsearch://reference/query-languages/esql/commands/ts.md) (in technical preview) to query time series data streams. The `TS` command is optimized for time series data. It also enables the use of aggregation functions that efficiently process metrics per time series, before aggregating results.
122114

123115

124116
## Next steps [tsds-whats-next]

0 commit comments

Comments
 (0)