diff --git a/manage-data/data-store/data-streams.md b/manage-data/data-store/data-streams.md index 98ea0e4d64..1e8071ed8f 100644 --- a/manage-data/data-store/data-streams.md +++ b/manage-data/data-store/data-streams.md @@ -17,6 +17,22 @@ A data stream acts as a layer of abstraction over a set of indices that are opti You can submit indexing and search requests directly to a data stream. The stream automatically routes the request to backing indices that store the stream’s data. You can use [{{ilm}} ({{ilm-init}})](../lifecycle/index-lifecycle-management.md) to automate the management of these backing indices. For example, you can use {{ilm-init}} to automatically move older backing indices to less expensive hardware and delete unneeded indices. {{ilm-init}} can help you reduce costs and overhead as your data grows. +You can also use a [data stream lifecycle](../lifecycle/data-stream.md) to automate lifecycle management according to your retention requirements. + +:::{admonition} Streams +:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} + +Starting with {{stack}} version 9.2, **Streams** provides a centralized interface for managing your data in {{kib}}. It consolidates common data management tasks and eliminates the need for manual configuration of multiple applications and components. A stream directly corresponds to an {{es}} data stream, for example `logs-myapp-default`. Any changes that you make in Streams are automatically propagated to the associated data stream. + +You can use Streams to perform the following data management tasks: +* defining parsing and field extraction logic to structure incoming data {applies_to}`"stack": "ga 9.2, preview 9.1"` +* configuring data retention policies {applies_to}`"stack": "ga 9.2, preview 9.1"` +* manually adjusting index settings {applies_to}`"stack": "ga 9.2, preview 9.1"` +* managing and updating field mappings {applies_to}`stack: ga 9.2` +* identifying failed and degraded documents {applies_to}`stack: ga 9.2` + +For more information, refer to [Streams](/solutions/observability/streams/streams.md). +::: ## Should you use a data stream? [should-you-use-a-data-stream] diff --git a/manage-data/data-store/data-streams/failure-store.md b/manage-data/data-store/data-streams/failure-store.md index 70a78d765d..9c8bf04d64 100644 --- a/manage-data/data-store/data-streams/failure-store.md +++ b/manage-data/data-store/data-streams/failure-store.md @@ -90,6 +90,12 @@ PUT _data_stream/my-datastream-existing/_options 1. Redirecting failed documents into the failure store will now be disabled. +:::{tip} +:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} + +You can also enable the data stream failure store in {{kib}}. Locate the data stream on the **Streams** page, where a stream directly corresponds to a data stream. Select a stream to view its details and go to the **Retention** tab where you can find the **Enable failure store** button. +::: + ### Enable failure store via cluster setting [set-up-failure-store-cluster-setting] If you have a large number of existing data streams you may want to enable their failure stores in one place. Instead of updating each of their options individually, set `data_streams.failure_store.enabled` to a list of index patterns in the [cluster settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). Any data streams that match one of these patterns will operate with their failure store enabled. diff --git a/manage-data/data-store/data-streams/manage-data-stream.md b/manage-data/data-store/data-streams/manage-data-stream.md index bb33d72eb7..b0db60c0e8 100644 --- a/manage-data/data-store/data-streams/manage-data-stream.md +++ b/manage-data/data-store/data-streams/manage-data-stream.md @@ -21,4 +21,24 @@ In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB ind * To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream’s name. From this view, you can navigate to **Discover** to further explore data within the data stream. * To view information about the stream’s backing indices, click the number in the **Indices** column. -* To modify the data retention value, select a data stream, open the **Manage** menu, and click **Edit data retention**. \ No newline at end of file +* To modify the data retention value, select a data stream, open the **Manage** menu, and click **Edit data retention**. + +## Manage data streams in Streams +```{applies_to} +serverless: ga +stack: preview 9.1, ga 9.2 +``` +Starting with {{stack}} version 9.2, [**Streams**](/solutions/observability/streams/streams.md) provides a centralized interface for managing your data in {{kib}}. It consolidates common data management tasks and eliminates the need for manual configuration of multiple applications and components. A stream directly corresponds to an {{es}} data stream, for example `logs-myapp-default`. Any changes that you make in Streams are automatically propagated to the associated data stream. + +:::{image} /manage-data/images/data-stream-management-streams.png +:alt: Streams details +:screenshot: +::: + + +You can perform the following data management tasks in Streams: +* [define parsing and field extraction logic](ADD LINK to /solutions/observability/streams/management/extract.md) to structure incoming data {applies_to}`"stack": "ga 9.2, preview 9.1"` +* [configure data retention policies](ADD LINK to /solutions/observability/streams/management/retention.md) {applies_to}`"stack": "ga 9.2, preview 9.1"` +* [manually adjust index settings](ADD LINK to/solutions/observability/streams/management/advanced.md) {applies_to}`"stack": "ga 9.2, preview 9.1"` +* [manage and update field mappings](ADD LINK to/solutions/observability/streams/management/schema.md) {applies_to}`stack: ga 9.2` +* [identify failed and degraded documents](ADD LINK to/solutions/observability/streams/management/data-quality.md) {applies_to}`stack: ga 9.2` diff --git a/manage-data/data-store/data-streams/modify-data-stream.md b/manage-data/data-store/data-streams/modify-data-stream.md index 56ebdb5622..48a48d2cf1 100644 --- a/manage-data/data-store/data-streams/modify-data-stream.md +++ b/manage-data/data-store/data-streams/modify-data-stream.md @@ -91,6 +91,11 @@ To add a mapping for a new field to a data stream, following these steps: ``` +:::{tip} +:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} + +You can also add a new field mapping to a data stream in {{kib}}. Locate the data stream on the **Streams** page, where a stream directly corresponds to a data stream. Select a stream to view its details and go to the **Schema** tab to add a new field. +::: ### Change an existing field mapping in a data stream [change-existing-field-mapping-in-a-data-stream] @@ -168,6 +173,12 @@ The documentation for each [mapping parameter](elasticsearch://reference/elastic ``` +:::{tip} +:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} + +You can also update an existing field's mapping in {{kib}}. Locate the data stream on the **Streams** page, where a stream directly corresponds to a data stream. Select the stream to view its details and go to the **Schema** tab to make your updates. For more information, refer to [ADD LINK to /solutions/observability/streams/management/schema.md when it's merged]. +::: + Except for supported mapping parameters, we don’t recommend you change the mapping or field data type of existing fields, even in a data stream’s matching index template or its backing indices. Changing the mapping of an existing field could invalidate any data that’s already indexed. If you need to change the mapping of an existing field, create a new data stream and reindex your data into it. See [Use reindex to change mappings or settings](../data-streams/modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). @@ -218,6 +229,14 @@ To change the `index.lifecycle.name` setting, first use the [remove policy API]( :::: +:::{tip} +:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} + +You can also update certain dynamic index settings for a data stream in {{kib}}, such as the number of shards, replicas, and the refresh interval used. + +Locate the data stream on the **Streams** page, where a stream directly corresponds to a data stream. Select the stream to view its details and go to the **Advanced** tab to make your adjustments. For more information, refer to [ADD LINK to /solutions/observability/streams/management/advanced.md#index-configuration when it's merged]. +::: + ### Change a static index setting for a data stream [change-static-index-setting-for-a-data-stream] [Static index settings](elasticsearch://reference/elasticsearch/index-settings/index.md) can only be set when a backing index is created. You cannot update static index settings using the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). diff --git a/manage-data/data-store/index-basics.md b/manage-data/data-store/index-basics.md index c47084d92d..e5bfcab7ef 100644 --- a/manage-data/data-store/index-basics.md +++ b/manage-data/data-store/index-basics.md @@ -153,6 +153,13 @@ In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB ind * To modify the data retention value, select a data stream, open the **Manage** menu, and click **Edit data retention**. On {{stack}}, this action is only available if your data stream is not managed by an ILM policy. * To view more information about a data stream including it's lifecycle settings, click the stream's name. +:::{admonition} Streams +:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} + +Starting with {{stack}} version 9.2, **Streams** provides a centralized interface for common data management tasks in {{kib}}, including tasks such as [modifying data retention](/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md#data-retention-streams) values. +For more information, refer to [Streams](/solutions/observability/streams/streams.md). +::: + ### Manage index templates [index-management-manage-index-templates] An index template is a type of [template](/manage-data/data-store/templates.md) that tells {{es}} how to configure an index when it is created. diff --git a/manage-data/images/data-stream-management-streams.png b/manage-data/images/data-stream-management-streams.png new file mode 100644 index 0000000000..403a0f0669 Binary files /dev/null and b/manage-data/images/data-stream-management-streams.png differ diff --git a/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md b/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md index db9a23816d..acb96d1eb6 100644 --- a/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md +++ b/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md @@ -10,7 +10,7 @@ products: # Tutorial: Configure data stream retention [tutorial-manage-data-stream-retention] -In this tutorial, we are going to go over the data stream lifecycle retention; we will define it, go over how it can be configured and how it can gets applied. Keep in mind, the following options apply only to data streams that are managed by the data stream lifecycle. +In this tutorial, we are going to go over the data stream lifecycle retention; we will define it, go over how it can be configured and how it gets applied. Keep in mind, the following options apply only to data streams that are managed by the data stream lifecycle. 1. [What is data stream retention?](#what-is-retention) 2. [How to configure retention?](#retention-configuration) @@ -42,6 +42,11 @@ The result should look like this: 2. Ensure that the lifecycle is enabled, meaning this should be `true`. +:::{tip} +:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} + +You can also review how a data stream is managed by locating it on the **Streams** page in {{kib}}. A stream directly corresponds to a data stream. Select a stream to view its details and go to the **Retention** tab. +::: ## What is data stream retention? [what-is-retention] @@ -67,9 +72,9 @@ Global default and max retention do not apply to data streams internal to elasti ## How to configure retention? [retention-configuration] -* By setting the `data_retention` on the data stream level. This retention can be configured in two ways: +* Configure data retention at the data stream level using the `data_retention` setting. You can do this in two ways: - — For new data streams, it can be defined in the index template that would be applied during the data stream’s creation. You can use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template), for example: + * For a new data stream, the `data_retention` setting can be included in the index template that is applied when the data stream is created. You can use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template), for example: ```console PUT _index_template/template @@ -88,7 +93,7 @@ Global default and max retention do not apply to data streams internal to elasti } ``` - — For an existing data stream, it can be set via the [PUT lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle). + * For an existing data stream, the `data_retention` setting can be configured using the [PUT lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle). ```console PUT _data_stream/my-data-stream/_lifecycle @@ -99,7 +104,13 @@ Global default and max retention do not apply to data streams internal to elasti 1. The retention period of this data stream is set to 30 days. -* By setting the global retention via the `data_streams.lifecycle.retention.default` and/or `data_streams.lifecycle.retention.max` that are set on a cluster level. You can be set via the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). For example: + :::{tip} + :applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} + + To adjust the retention period of a data stream in {{kib}}, locate a data stream on the **Streams** page. A stream directly corresponds to a data stream. Next, select a stream to view its details and review the **Retention** tab to find out how it's managed before making your adjustments. + ::: + +* By setting the global retention via the `data_streams.lifecycle.retention.default` and/or `data_streams.lifecycle.retention.max` that are set on a cluster level. You can set this via the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). For example: ```console PUT /_cluster/settings diff --git a/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md index 4d03aaca34..f0f55fb756 100644 --- a/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md @@ -13,7 +13,14 @@ products: In this tutorial we’ll look at migrating an existing data stream from [Index Lifecycle Management ({{ilm-init}})](../index-lifecycle-management.md) to [data stream lifecycle](../data-stream.md). The existing {{ilm-init}} managed backing indices will continue to be managed by {{ilm-init}} until they age out and get deleted by {{ilm-init}}; however, the new backing indices will be managed by data stream lifecycle. This way, a data stream is gradually migrated away from being managed by {{ilm-init}} to being managed by data stream lifecycle. As we’ll see, {{ilm-init}} and data stream lifecycle can co-manage a data stream; however, an index can only be managed by one system at a time. -To migrate a data stream from {{ilm-init}} to data stream lifecycle we’ll have to execute two steps: +:::{admonition} Configure data retention policies in Streams +:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} + +Starting with {{stack}} version 9.2, [Streams](/solutions/observability/streams/streams.md) provides a centralized interface for common data management tasks in {{kib}}, including tasks such as configuring data retention policies. You can choose to retain your data indefinitely, for a custom period, or by following an existing ILM policy. For more information, refer to [Manage data retention in Streams](/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md#data-retention-streams). + +::: + +To migrate a data stream from {{ilm-init}} to data stream lifecycle using APIs we’ll have to execute two steps: 1. Update the index template that’s backing the data stream to set [prefer_ilm](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle. 2. Configure the data stream lifecycle for the *existing* data stream using the [lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle). diff --git a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md index 89e0bcabeb..a3503df086 100644 --- a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md +++ b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md @@ -14,6 +14,7 @@ Follow these steps to configure or remove data stream lifecycle settings for an - [Set a data stream’s lifecycle](#set-lifecycle) - [Remove the lifecycle for a data stream](#delete-lifecycle) +- [Manage data retention in Streams](#data-retention-streams) Note that these steps are for data stream lifecycle only. For the steps to configure {{ilm}}, refer to the [{{ilm-init}} documentation](/manage-data/lifecycle/index-lifecycle-management.md). For a comparison between the two, refer to [](/manage-data/lifecycle.md). @@ -209,4 +210,19 @@ GET .ds-my-data-stream-*/_lifecycle/explain 3. The name of another backing index. 4. Indication that the index is not managed by the data stream lifecycle. ::: -::::: \ No newline at end of file +::::: + +## Manage data retention in Streams [data-retention-streams] +```{applies_to} +serverless: ga +stack: preview 9.1, ga 9.2 +``` + +Starting with {{stack}} version 9.2, **Streams** provides a centralized interface for common data management tasks in {{kib}}. + +1. Go to the **Streams** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). +1. A stream directly corresponds to an {{es}} data stream. Select a stream to view its details. +1. Go to the **Retention** tab to set how long your stream retains data and to get insight into your stream's data ingestion and storage size. +1. Select **Edit data retention** and choose to retain your data indefinitely, for a custom period, or by following an existing ILM policy. + +Any changes you make in Streams are automatically propagated to the associated data stream. diff --git a/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md b/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md index 132efe3ad0..2d3aac6b95 100644 --- a/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md +++ b/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md @@ -45,6 +45,16 @@ To learn how to create a specialized ILM policy for any data stream, such as tho 1. Select the data stream to view details. The flyout that opens includes direct links to the ILM policy and the index template. ![Data stream status page](/manage-data/images/elasticsearch-reference-datastream-status.png "") + + +**To view the current lifecycle status for a datastream in Streams:** {applies_to}`"stack": "ga 9.2, preview 9.1"` + +Starting with {{stack}} version 9.2, [Streams](/solutions/observability/streams/streams.md) provides a centralized interface for common data management tasks in {{kib}}. + +1. Go to the **Streams** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). +1. A stream directly corresponds to an {{es}} data stream. Select a stream to view its details. +1. Go to the **Retention** tab to see how long your stream retains data and to get insight into your stream's data ingestion and storage size. A stream can retain the data indefinitely, for a custom period, or by following an existing ILM policy. + :::: :::{tab-item} API