diff --git a/manage-data/lifecycle/data-stream.md b/manage-data/lifecycle/data-stream.md index 4d239e40b5..ca352b1f3e 100644 --- a/manage-data/lifecycle/data-stream.md +++ b/manage-data/lifecycle/data-stream.md @@ -48,11 +48,12 @@ Steps `2-4` apply only to backing indices that are not already managed by {{ilm- Since the lifecycle is configured on the data stream level, the process to configure a lifecycle on a new data stream and on an existing one differ. -In the following sections, we will go through the following tutorials: +Four tutorials are available to help you set up and manage data streams with data stream lifecycle: -* To create a new data stream with a lifecycle, you need to add the data stream lifecycle as part of the index template that matches the name of your data stream (see [Tutorial: Create a data stream with a lifecycle](data-stream/tutorial-create-data-stream-with-lifecycle.md)). When a write operation with the name of your data stream reaches {{es}} then the data stream will be created with the respective data stream lifecycle. -* To update the lifecycle of an existing data stream you need to use the [data stream lifecycle APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-data-stream) to edit the lifecycle on the data stream itself (see [Tutorial: Update existing data stream](data-stream/tutorial-update-existing-data-stream.md)). -* Migrate an existing {{ilm-init}} managed data stream to Data stream lifecycle using [Tutorial: Migrate ILM managed data stream to data stream lifecycle](data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md). +* To create a new data stream with a lifecycle, add the data stream lifecycle as part of the index template that matches the name of your data stream. See [Tutorial: Create a data stream with a lifecycle](data-stream/tutorial-create-data-stream-with-lifecycle.md) for the detailed steps. When a write operation with the name of your data stream reaches {{es}} then the data stream will be created with the respective data stream lifecycle. +* To update the lifecycle settings for an individual, existing data stream, use the [data stream lifecycle APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-data-stream). See [Tutorial: Update existing data stream](data-stream/tutorial-update-existing-data-stream.md) for details. +* Retention settings for data streams can be configured both individually, at the data stream level, and globally, for all data streams in a cluster. To learn more, refer to [Tutorial: Configure data stream retention](/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md). +* To migrate an existing {{ilm-init}} managed data stream to data stream lifecycle, follow the steps in [Tutorial: Migrate ILM managed data stream to data stream lifecycle](data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md). ::::{note} Updating the data stream lifecycle of an existing data stream is different from updating the settings or the mapping, because it is applied on the data stream level and not on the individual backing indices. diff --git a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md index f9808f44a9..6677096b5a 100644 --- a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md @@ -30,7 +30,7 @@ You can use the [create index template API](https://www.elastic.co/docs/api/doc/ ```console PUT _index_template/my-index-template { - "index_patterns": ["my-data-stream*"], + "index_patterns": ["my-data-stream-test"], <1> "data_stream": { }, "priority": 500, "template": { @@ -43,7 +43,7 @@ PUT _index_template/my-index-template } } ``` - +1. In this case the index template will be applied to a data stream named `my-data-stream-test`. You can optionally use a wildcard (`*`) in the index pattern to match all data streams created (either manually or using an indexing request) that have a name matching the specified pattern. ## Create a data stream [create-data-stream-with-lifecycle] @@ -52,13 +52,13 @@ You can create a data stream in two ways: 1. By manually creating the stream using the [create data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream). The stream’s name must still match one of your template’s index patterns. ```console - PUT _data_stream/my-data-stream + PUT _data_stream/my-data-stream-test ``` 2. By [indexing requests](../../data-store/data-streams/use-data-stream.md#add-documents-to-a-data-stream) that target the stream’s name. This name must match one of your index template’s index patterns. ```console - PUT my-data-stream/_bulk + PUT my-data-stream-test/_bulk { "create":{ } } { "@timestamp": "2099-05-06T16:21:15.000Z", "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736" } { "create":{ } } @@ -72,7 +72,7 @@ You can create a data stream in two ways: You can use the [get data stream lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-lifecycle) to see the data stream lifecycle of your data stream and the [explain data stream lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-explain-data-lifecycle) to see the exact state of each backing index. ```console -GET _data_stream/my-data-stream/_lifecycle +GET _data_stream/my-data-stream-test/_lifecycle ``` The result will look like this: @@ -81,7 +81,7 @@ The result will look like this: { "data_streams": [ { - "name": "my-data-stream", <1> + "name": "my-data-stream-test", <1> "lifecycle": { "enabled": true, <2> "data_retention": "7d", <3> @@ -103,16 +103,20 @@ The result will look like this: If you want to see more information about how the data stream lifecycle is applied on individual backing indices use the [explain data stream lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-explain-data-lifecycle): ```console -GET .ds-my-data-stream-*/_lifecycle/explain +GET .ds-my-data-stream-test/_lifecycle/explain ``` +:::{tip} +You can use a wildcard (`*`) in the data stream name to retrieve the lifecycle status for all data streams matching the pattern. +::: + The result will look like this: ```console-result { "indices": { - ".ds-my-data-stream-2023.04.19-000001": { - "index": ".ds-my-data-stream-2023.04.19-000001", <1> + ".ds-my-data-stream-test-2023.04.19-000001": { + "index": ".ds-my-data-stream-test-2023.04.19-000001", <1> "managed_by_lifecycle": true, <2> "index_creation_date_millis": 1681918009501, "time_since_index_creation": "1.6m", <3> diff --git a/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md b/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md index a7c8df6e41..db9a23816d 100644 --- a/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md +++ b/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md @@ -8,7 +8,7 @@ products: - id: elasticsearch --- -# Tutorial: Data stream retention [tutorial-manage-data-stream-retention] +# Tutorial: Configure data stream retention [tutorial-manage-data-stream-retention] In this tutorial, we are going to go over the data stream lifecycle retention; we will define it, go over how it can be configured and how it can gets applied. Keep in mind, the following options apply only to data streams that are managed by the data stream lifecycle. diff --git a/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md index 682446c623..55fee8e505 100644 --- a/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md @@ -312,8 +312,6 @@ GET _data_stream/dsl-data-stream 3. The new write index received the `false` value for the `prefer_ilm` setting, as we configured in the index template 4. The new write index is managed by `Data stream lifecycle` - - ## Migrate data stream back to ILM [migrate-from-dsl-to-ilm] We can easily change this data stream to be managed by {{ilm-init}} because we didn’t remove the {{ilm-init}} policy when we [updated the index template](#update-index-template-for-dsl). @@ -333,7 +331,7 @@ PUT _data_stream/dsl-data-stream/_lifecycle } ``` -1. The `enabled` flag can be ommitted and defaults to `true` however, here we explicitly configure it to `false` Let’s check the state of the data stream: +1. The `enabled` flag can be omitted and defaults to `true` however, here we explicitly configure it to `false` Let’s check the state of the data stream: ```console diff --git a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md index d3ff700aaa..89e0bcabeb 100644 --- a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md +++ b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md @@ -8,13 +8,14 @@ products: - id: elasticsearch --- -# Tutorial: Update existing data stream [tutorial-manage-existing-data-stream] +# Tutorial: Update the lifecycle of a data stream [tutorial-manage-existing-data-stream] -To update the lifecycle of an existing data stream, perform the following actions: +Follow these steps to configure or remove data stream lifecycle settings for an existing, individual data stream: -1. [Set a data stream’s lifecycle](#set-lifecycle) -2. [Remove lifecycle for a data stream](#delete-lifecycle) +- [Set a data stream’s lifecycle](#set-lifecycle) +- [Remove the lifecycle for a data stream](#delete-lifecycle) +Note that these steps are for data stream lifecycle only. For the steps to configure {{ilm}}, refer to the [{{ilm-init}} documentation](/manage-data/lifecycle/index-lifecycle-management.md). For a comparison between the two, refer to [](/manage-data/lifecycle.md). ## Set a data stream’s lifecycle [set-lifecycle] @@ -143,7 +144,7 @@ The response will look like: ::: ::::: -## Remove lifecycle for a data stream [delete-lifecycle] +## Remove the lifecycle for a data stream [delete-lifecycle] To remove the lifecycle of a data stream you can use the **Index Management** tools in {{kib}} or the {{es}} [delete lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-lifecycle).