From 919776d4371803d93823ec1dad491ce65bd5e4fe Mon Sep 17 00:00:00 2001 From: Vlada Chirmicci Date: Fri, 24 Oct 2025 16:29:24 +0100 Subject: [PATCH 1/6] Update title for the built-in ILM policies customization topic Fixes #3285 to clarify that the scope of the tutorial refers to Elastic Agent data streams that use built-in ILM policies. --- manage-data/lifecycle/index-lifecycle-management.md | 2 +- .../index-lifecycle-management/configure-lifecycle-policy.md | 2 +- .../manage-lifecycle-integrations-data.md | 2 +- .../lifecycle/index-lifecycle-management/policy-view-status.md | 2 +- .../tutorial-customize-built-in-policies.md | 3 ++- 5 files changed, 6 insertions(+), 5 deletions(-) diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index ea786fa54a..4a31c262f7 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -78,7 +78,7 @@ You can create and manage index lifecycle policies through {{kib}}'s [Index Mana * [View the lifecycle status of an index or datastream](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md) * [Update or switch a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md) * [Restore a managed data stream or index](/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md) -* [Customize built-in policies](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) +* [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) Default {{ilm}} policies are created automatically when you install an [Elastic Integration](integration-docs://reference/index.md), or when you use {{agent}}, {{beats}}, or the {{ls}} {{es}} output plugin to send data to the {{stack}}. diff --git a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md index 536fcd6e6e..f2f305b3a1 100644 --- a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md +++ b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md @@ -220,7 +220,7 @@ The name of the index must match the pattern defined in the index template and e This step is required only when you're planning to use {{ilm-init}} with rolling indices. It is not required when you're using data streams, where the initial managed index is created automatically. ::::{important} -When you enable {{ilm}} for {{beats}}, {{agent}}, or for the {{agent}} or {{ls}} {{es}} output plugins, the necessary policies and configuration changes are applied automatically. If you'd like to create a specialized ILM policy for any data stream, refer to our tutorial [Customize built-in policies](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md). +When you enable {{ilm}} for {{beats}}, {{agent}}, or for the {{agent}} or {{ls}} {{es}} output plugins, the necessary policies and configuration changes are applied automatically. If you'd like to create a specialized ILM policy for any data stream, refer to our tutorial [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md). :::: ::::{tab-set} diff --git a/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md b/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md index 906710b674..6d7f7020e0 100644 --- a/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md +++ b/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md @@ -61,5 +61,5 @@ For any data stream that you're interested in, you can [view its current lifecyc After you've identified one or more data streams for which you'd like to customize how the data is managed over time, refer to our tutorials: -* For a general guide about configuring a custom ILM policy for any managed data stream, try out our [Customize built-in policies](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) tutorial in the data lifecycle documentation. +* For a general guide about configuring a custom ILM policy for any managed data stream, try out our [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) tutorial in the data lifecycle documentation. * For the steps to customize an ILM policy for a set of data streams, such as all logs or metrics data streams across all namespaces, across only a selected namespace, and others, check the set of tutorials in [Customize data retention policies](/reference/fleet/data-streams-ilm-tutorial.md) in the {{fleet}} and {{agent}} reference documentation. diff --git a/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md b/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md index 27dbaa1de8..ae9d24a59d 100644 --- a/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md +++ b/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md @@ -36,7 +36,7 @@ If you're investigating an {{ilm-init}}-related problem, refer to [Troubleshoot :::{tip} {{es}} comes with many built-in ILM policies. For standard Observability or Security use cases, you will have two {{ilm-init}} policies configured automatically: `logs@lifecycle` for logs and `metrics@lifecycle` for metrics. -To learn how to create a specialized ILM policy for any data stream, such as those created when you install an Elastic Integration, refer to our tutorial [Customize built-in policies](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md). +To learn how to create a specialized ILM policy for any data stream, such as those created when you install an Elastic Integration, refer to our tutorial [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md). ::: **To view the current lifecycle status for a datastream:** diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md index 39ff1cc907..b62858dc09 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md @@ -1,4 +1,5 @@ --- +navigation_title: Customize built-in {{ilm-init}} policies mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/example-using-index-lifecycle-policy.html applies_to: @@ -7,7 +8,7 @@ products: - id: elasticsearch --- -# Customize built-in policies +# Customize built-in {{ilm-init}} policies for {{agent}} data streams {{es}} includes a set of built-in {{ilm-init}} policies that govern how managed indices transition as they age. This guide demonstrates how you can customize the lifecycle of a managed index, to adjust how the index transitions across [data tiers](/manage-data/lifecycle/data-tiers.md) and what [actions](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md#ilm-phase-actions), such as downsampling or shrinking, are performed on the index during each lifecycle phase. From 57a0702e485206fa30c433a276dcb4bfc20d1338 Mon Sep 17 00:00:00 2001 From: Vlada Chirmicci Date: Tue, 28 Oct 2025 10:22:23 +0000 Subject: [PATCH 2/6] Update title to emphasize you're modifying duplicate policies not the actual built-in ones --- .../tutorial-customize-built-in-policies.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md index b62858dc09..f3ae0ef8c5 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md @@ -1,5 +1,4 @@ --- -navigation_title: Customize built-in {{ilm-init}} policies mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/example-using-index-lifecycle-policy.html applies_to: @@ -8,7 +7,7 @@ products: - id: elasticsearch --- -# Customize built-in {{ilm-init}} policies for {{agent}} data streams +# Customize duplicates of built-in {{ilm-init}} policies {{es}} includes a set of built-in {{ilm-init}} policies that govern how managed indices transition as they age. This guide demonstrates how you can customize the lifecycle of a managed index, to adjust how the index transitions across [data tiers](/manage-data/lifecycle/data-tiers.md) and what [actions](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md#ilm-phase-actions), such as downsampling or shrinking, are performed on the index during each lifecycle phase. From fd3c8c017f2c1dfafcbd33474290a561ea69ab20 Mon Sep 17 00:00:00 2001 From: Vlada Chirmicci Date: Tue, 28 Oct 2025 15:06:31 +0000 Subject: [PATCH 3/6] Update link text to clarify scope of tutorial --- .../manage-lifecycle-integrations-data.md | 2 +- .../lifecycle/index-lifecycle-management/policy-view-status.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md b/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md index 6d7f7020e0..fd0501d8cd 100644 --- a/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md +++ b/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md @@ -61,5 +61,5 @@ For any data stream that you're interested in, you can [view its current lifecyc After you've identified one or more data streams for which you'd like to customize how the data is managed over time, refer to our tutorials: -* For a general guide about configuring a custom ILM policy for any managed data stream, try out our [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) tutorial in the data lifecycle documentation. +* For steps to modify {{ilm-init}} policies without changing the built-in defaults, try out the [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) tutorial in the data lifecycle documentation. * For the steps to customize an ILM policy for a set of data streams, such as all logs or metrics data streams across all namespaces, across only a selected namespace, and others, check the set of tutorials in [Customize data retention policies](/reference/fleet/data-streams-ilm-tutorial.md) in the {{fleet}} and {{agent}} reference documentation. diff --git a/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md b/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md index ae9d24a59d..198b88b6c9 100644 --- a/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md +++ b/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md @@ -36,7 +36,7 @@ If you're investigating an {{ilm-init}}-related problem, refer to [Troubleshoot :::{tip} {{es}} comes with many built-in ILM policies. For standard Observability or Security use cases, you will have two {{ilm-init}} policies configured automatically: `logs@lifecycle` for logs and `metrics@lifecycle` for metrics. -To learn how to create a specialized ILM policy for any data stream, such as those created when you install an Elastic Integration, refer to our tutorial [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md). +To learn how to create and adjust copies of built-in {{ilm-init}} policies for managed data streams, such as those created when you install an Elastic Integration, refer to our tutorial [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md). ::: **To view the current lifecycle status for a datastream:** From c036bfacdeba6d20dd0c7758fb0c12bbe7d8d1b1 Mon Sep 17 00:00:00 2001 From: Vlada Chirmicci Date: Wed, 29 Oct 2025 12:22:08 +0000 Subject: [PATCH 4/6] Adding more context around the agent example and applicability for modifying other policy duplicates --- .../tutorial-customize-built-in-policies.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md index f3ae0ef8c5..87536a0a1b 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md @@ -9,11 +9,11 @@ products: # Customize duplicates of built-in {{ilm-init}} policies -{{es}} includes a set of built-in {{ilm-init}} policies that govern how managed indices transition as they age. This guide demonstrates how you can customize the lifecycle of a managed index, to adjust how the index transitions across [data tiers](/manage-data/lifecycle/data-tiers.md) and what [actions](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md#ilm-phase-actions), such as downsampling or shrinking, are performed on the index during each lifecycle phase. +{{es}} includes a set of built-in {{ilm-init}} policies that define how managed indices transition across [data tiers](/manage-data/lifecycle/data-tiers.md) and what [actions](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md#ilm-phase-actions), such as rollover, downsampling, or shrinking, are performed at each phase. -Setting a custom {{ilm-init}} policy is useful when you have a specific set of indices, for example a set of Kubernetes logs which can grow to be quite large in volume, for which you don't want to use the default data retention duration and other {{ilm-init}} settings. +This tutorial demonstrates how to create a customized copy of a built-in {{ilm-init}} policy to better fit your data retention, performance, or storage requirements. You should never edit managed policies directly, because updates to {{es}} or Elastic integrations might overwrite those changes. Instead, you can duplicate a built-in policy, modify the duplicate, and assign it to your index or component templates. -[{{agent}}](/reference/fleet/index.md) uses the following set of built-in {{ilm-init}} policies to manage backing indices for its data streams: +While this tutorial uses [{{agent}}](/reference/fleet/index.md) and its built-in `logs@lifecycle` policy as an example, the same process can be applied to any built-in policies. Common examples include: * `logs@lifecycle` * `logs-otel@lifecycle` @@ -23,7 +23,11 @@ Setting a custom {{ilm-init}} policy is useful when you have a specific set of i * `traces@lifecycle` * `traces-otel@lifecycle` -This tutorial covers customizing the way ingested logging data is managed. Rather than use the default lifecycle settings from the built-in `logs@lifecycle` {{ilm-init}} policy, you can use the **Index Lifecycle Policies** feature in {{kib}} to tailor a new policy based on your application’s specific performance, resilience, and retention requirements. This involves three main steps: +Customizing an {{ilm-init}} policy is useful when you have specific data retention or rollover requirements. For example, large log or metrics data streams might need different retention periods than the built-in defaults. + +This tutorial covers customizing the way ingested logging data is managed. Rather than use the default lifecycle settings from the built-in `logs@lifecycle` {{ilm-init}} policy, you can use the **Index Lifecycle Policies** feature in {{kib}} to tailor a new policy based on your application’s specific performance, resilience, and retention requirements. You can adapt the same steps for any policy that manages your data streams. + +The process involves three main steps: 1. [Create a duplicate of the `logs@lifecycle` policy](#example-using-index-lifecycle-policy-duplicate-ilm-policy). 2. [Modify the new policy to suit your requirements](#ilm-ex-modify-policy). 3. [Apply the new policy to your log data using a `logs@custom` component template](#example-using-index-lifecycle-policy-apply-policy). From f089d3d095e5193d8cd629161eee20016d6ac337 Mon Sep 17 00:00:00 2001 From: Vlada Chirmicci Date: Wed, 29 Oct 2025 16:43:36 +0000 Subject: [PATCH 5/6] Edits to make better link distinction in the integrations data doc --- .../manage-lifecycle-integrations-data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md b/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md index fd0501d8cd..7b3282e0a7 100644 --- a/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md +++ b/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md @@ -61,5 +61,5 @@ For any data stream that you're interested in, you can [view its current lifecyc After you've identified one or more data streams for which you'd like to customize how the data is managed over time, refer to our tutorials: -* For steps to modify {{ilm-init}} policies without changing the built-in defaults, try out the [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) tutorial in the data lifecycle documentation. +* For steps to customize an {{ilm-init}} policy for a data stream, try out the [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) tutorial in the data lifecycle documentation. * For the steps to customize an ILM policy for a set of data streams, such as all logs or metrics data streams across all namespaces, across only a selected namespace, and others, check the set of tutorials in [Customize data retention policies](/reference/fleet/data-streams-ilm-tutorial.md) in the {{fleet}} and {{agent}} reference documentation. From 8342752888513b6c16978d62558e9f094d228257 Mon Sep 17 00:00:00 2001 From: Vlada Chirmicci Date: Tue, 4 Nov 2025 14:20:09 +0000 Subject: [PATCH 6/6] Add statement about applicability to new indices + rollover to begin using the new policy immediately --- .../tutorial-customize-built-in-policies.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md index 87536a0a1b..2dd41cb91a 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md @@ -32,6 +32,8 @@ The process involves three main steps: 2. [Modify the new policy to suit your requirements](#ilm-ex-modify-policy). 3. [Apply the new policy to your log data using a `logs@custom` component template](#example-using-index-lifecycle-policy-apply-policy). +Once applied, your customized policy will govern any new indices created after the change. Existing indices will continue to use their current lifecycle policy until they roll over. If you want the policy to take effect immediately, you can manually [roll over](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover) the data stream. + :::{tip} * If you're using [Elastic integrations](https://docs.elastic.co/en/integrations) and are not yet familiar with which data streams are associated with them, refer to [Manage the lifecycle policy for integrations data](/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md).