Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8836b54
Adding Streams information into the Manage data section
yetanothertw Oct 8, 2025
2996bbb
Changed section to admonition
yetanothertw Oct 8, 2025
3c706c5
adding more applies_to tags
yetanothertw Oct 8, 2025
dc81d2c
trying diff tagging
yetanothertw Oct 8, 2025
55e9940
another attempt to add multiple tags in note
yetanothertw Oct 8, 2025
6ab5411
fixing tags per Fabri's example
yetanothertw Oct 8, 2025
7a2c96d
Add section about managing data retention in Streams
yetanothertw Oct 8, 2025
4eaa1f3
minor update to include new section in the tutorial context
yetanothertw Oct 8, 2025
62ed5a6
Updating the configure data stream retention tutorial
yetanothertw Oct 9, 2025
f2d0f94
Adding mentions of streams in the failure store for data stream doc
yetanothertw Oct 9, 2025
ea3f6f2
minor edits fot the modify ds page
yetanothertw Oct 10, 2025
b8a2f4e
Minor improvements suggested by David
yetanothertw Oct 10, 2025
e48257a
Adding steps to view a data stream's lifecycle in streams
yetanothertw Oct 10, 2025
cce95b3
add link to streams
yetanothertw Oct 10, 2025
44f28f5
Adding a section for managing data streams in Streams
yetanothertw Oct 10, 2025
380e50a
Merge branch 'main' into streams-274-md
yetanothertw Oct 10, 2025
a4cd59c
Add note about managing data retention in stream to the remaining DLM…
yetanothertw Oct 10, 2025
7722d3c
Merge branch 'main' into streams-274-md
yetanothertw Oct 20, 2025
d81a06a
Making some final revisions
yetanothertw Oct 21, 2025
b886395
Review feedback
yetanothertw Oct 22, 2025
7416006
Merge branch 'main' into streams-274-md
yetanothertw Oct 22, 2025
a32641a
Add links to the Obs content
yetanothertw Oct 22, 2025
2acaa78
One more link
yetanothertw Oct 22, 2025
6d981d0
Fix link to anchor
yetanothertw Oct 22, 2025
87908b8
Merge branch 'main' into streams-274-md
yetanothertw Oct 22, 2025
2056edb
Merge branch 'main' into streams-274-md
yetanothertw Oct 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions manage-data/data-store/data-streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,22 @@ A data stream acts as a layer of abstraction over a set of indices that are opti

You can submit indexing and search requests directly to a data stream. The stream automatically routes the request to backing indices that store the stream’s data. You can use [{{ilm}} ({{ilm-init}})](../lifecycle/index-lifecycle-management.md) to automate the management of these backing indices. For example, you can use {{ilm-init}} to automatically move older backing indices to less expensive hardware and delete unneeded indices. {{ilm-init}} can help you reduce costs and overhead as your data grows.

You can also use a [data stream lifecycle](../lifecycle/data-stream.md) to automate lifecycle management according to your retention requirements.

:::{admonition} Streams
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}

Starting with {{stack}} version 9.2, **Streams** provides a centralized interface for managing your data in {{kib}}. It consolidates common data management tasks and eliminates the need for manual configuration of multiple applications and components. A stream directly corresponds to an {{es}} data stream, for example `logs-myapp-default`. Any changes that you make in Streams are automatically propagated to the associated data stream.

You can use Streams to perform the following data management tasks:
* defining parsing and field extraction logic to structure incoming data {applies_to}`"stack": "ga 9.2, preview 9.1"`
* configuring data retention policies {applies_to}`"stack": "ga 9.2, preview 9.1"`
* manually adjusting index configuration settings {applies_to}`"stack": "ga 9.2, preview 9.1"`
* managing and updating field mappings {applies_to}`stack: ga 9.2`
* identifying failed and degraded documents {applies_to}`stack: ga 9.2`

For more information, refer to [Streams](/solutions/observability/streams/streams.md).
:::

## Should you use a data stream? [should-you-use-a-data-stream]

Expand Down
7 changes: 7 additions & 0 deletions manage-data/data-store/index-basics.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,13 @@ In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB ind
* To modify the data retention value, select a data stream, open the **Manage** menu, and click **Edit data retention**. On {{stack}}, this action is only available if your data stream is not managed by an ILM policy.
* To view more information about a data stream including it's lifecycle settings, click the stream's name.

:::{admonition} Streams
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}

Starting with {{stack}} version 9.2, **Streams** provides a centralized interface for common data management tasks in {{kib}}, including tasks such as [modifying data retention](/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md#data-retention-streams) values.
For more information, refer to [Streams](/solutions/observability/streams/streams.md).
:::

### Manage index templates [index-management-manage-index-templates]

An index template is a type of [template](/manage-data/data-store/templates.md) that tells {{es}} how to configure an index when it is created.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ products:

# Tutorial: Configure data stream retention [tutorial-manage-data-stream-retention]

In this tutorial, we are going to go over the data stream lifecycle retention; we will define it, go over how it can be configured and how it can gets applied. Keep in mind, the following options apply only to data streams that are managed by the data stream lifecycle.
In this tutorial, we are going to go over the data stream lifecycle retention; we will define it, go over how it can be configured and how it gets applied. Keep in mind, the following options apply only to data streams that are managed by the data stream lifecycle.

1. [What is data stream retention?](#what-is-retention)
2. [How to configure retention?](#retention-configuration)
Expand Down Expand Up @@ -42,6 +42,11 @@ The result should look like this:
2. Ensure that the lifecycle is enabled, meaning this should be `true`.


:::{tip}
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}

You can also review how a data stream is managed by locating it on the **Streams** page in {{kib}}. A stream directly corresponds to a data stream. Select a stream to view its details and go to the **Retention** tab.
:::

## What is data stream retention? [what-is-retention]

Expand Down Expand Up @@ -69,7 +74,7 @@ Global default and max retention do not apply to data streams internal to elasti

* By setting the `data_retention` on the data stream level. This retention can be configured in two ways:

For new data streams, it can be defined in the index template that would be applied during the data stream’s creation. You can use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template), for example:
* For new data streams, it can be defined in the index template that would be applied during the data stream’s creation. You can use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template), for example:

```console
PUT _index_template/template
Expand All @@ -88,7 +93,7 @@ Global default and max retention do not apply to data streams internal to elasti
}
```

For an existing data stream, it can be set via the [PUT lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle).
* For an existing data stream, it can be set via the [PUT lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle).

```console
PUT _data_stream/my-data-stream/_lifecycle
Expand All @@ -99,7 +104,13 @@ Global default and max retention do not apply to data streams internal to elasti

1. The retention period of this data stream is set to 30 days.

* By setting the global retention via the `data_streams.lifecycle.retention.default` and/or `data_streams.lifecycle.retention.max` that are set on a cluster level. You can be set via the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). For example:
:::{tip}
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}

To adjust the retention period of a data stream in {{kib}}, locate a data stream on the **Streams** page. A stream directly corresponds to a data stream. Next, select a stream to view its details and review the **Retention** tab to find out how it's managed before making your adjustments.
:::

* By setting the global retention via the `data_streams.lifecycle.retention.default` and/or `data_streams.lifecycle.retention.max` that are set on a cluster level. You can set this via the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). For example:

```console
PUT /_cluster/settings
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Follow these steps to configure or remove data stream lifecycle settings for an

- [Set a data stream’s lifecycle](#set-lifecycle)
- [Remove the lifecycle for a data stream](#delete-lifecycle)
- [Manage data retention in Streams](#data-retention-streams)

Note that these steps are for data stream lifecycle only. For the steps to configure {{ilm}}, refer to the [{{ilm-init}} documentation](/manage-data/lifecycle/index-lifecycle-management.md). For a comparison between the two, refer to [](/manage-data/lifecycle.md).

Expand Down Expand Up @@ -209,4 +210,19 @@ GET .ds-my-data-stream-*/_lifecycle/explain
3. The name of another backing index.
4. Indication that the index is not managed by the data stream lifecycle.
:::
:::::
:::::

## Manage data retention in Streams [data-retention-streams]
```{applies_to}
serverless: ga
stack: preview 9.1, ga 9.2
```

Starting with {{stack}} version 9.2, **Streams** provides a centralized interface for common data management tasks in {{kib}}.

1. Go to the **Streams** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
1. A stream directly corresponds to an {{es}} data stream. Select a stream to view its details.
1. Go to the **Retention** tab to set how long your stream retains data and to get insight into your stream's data ingestion and storage size.
1. Select **Edit data retention** and choose to retain your data indefinitely, for a custom period, or by following an existing ILM policy.

Any changes you make in Streams are automatically propagated to the associated data stream.
Loading