Skip to content

ILM docs: improve @custom component template usage/details #3511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Oct 20, 2025
Merged

ILM docs: improve @custom component template usage/details #3511

Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
99e45cb
ILM docs: link to better @custom component template details
kilfoyle Oct 16, 2025
0030787
fixup
kilfoyle Oct 16, 2025
bf31c71
add note about k8s
kilfoyle Oct 16, 2025
c3aeda6
Clear up @custom stuff; add screencap
kilfoyle Oct 17, 2025
d7cd4de
Update manage-data/lifecycle/index-lifecycle-management/tutorial-cust…
kilfoyle Oct 20, 2025
4401a56
Merge branch 'main' into 1557/custom-logs
kilfoyle Oct 20, 2025
d8155b8
Merge branch 'main' into 1557/custom-logs
kilfoyle Oct 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions ...ta/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ products:

{{es}} includes a set of built-in {{ilm-init}} policies that govern how managed indices transition as they age. This guide demonstrates how you can customize the lifecycle of a managed index, to adjust how the index transitions across [data tiers](/manage-data/lifecycle/data-tiers.md) and what [actions](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md#ilm-phase-actions), such as downsampling or shrinking, are performed on the index during each lifecycle phase.

Setting a custom {{ilm-init}} policy is useful when you have a specific set of indices, for example a set of Kubernetes logs which can grow to be quite large in volume, for which you don't want to use the default data retention duration and other {{ilm-init}} settings.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

beautiful :)


[{{agent}}](/reference/fleet/index.md) uses the following set of built-in {{ilm-init}} policies to manage backing indices for its data streams:

* `logs@lifecycle`
Expand All @@ -27,9 +29,11 @@ This tutorial covers customizing the way ingested logging data is managed. Rathe
3. [Apply the new policy to your log data using a `logs@custom` component template](#example-using-index-lifecycle-policy-apply-policy).

:::{tip}
If you're using [Elastic integrations](https://docs.elastic.co/en/integrations) and are not yet familiar with which data streams are associated with them, refer to [Manage the lifecycle policy for integrations data](/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md).
* If you're using [Elastic integrations](https://docs.elastic.co/en/integrations) and are not yet familiar with which data streams are associated with them, refer to [Manage the lifecycle policy for integrations data](/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md).

* If you're looking for a more advanced use case, such as customizing an ILM policy for a selected set of data streams in one or more integrations or namespaces, check the set of tutorials in [Customize data retention policies](/reference/fleet/data-streams-ilm-tutorial.md) in the {{fleet}} and {{agent}} reference documentation.

If you're looking for a more advanced use case, such as customizing an ILM policy for a selected set of data streams in one or more integrations or namespaces, check the set of tutorials in [Customize data retention policies](/reference/fleet/data-streams-ilm-tutorial.md) in the {{fleet}} and {{agent}} reference documentation.
These tutorials go into greater depth about creating and using `@custom` component templates. For example, the tutorial [Apply an ILM policy to all data streams generated from {{fleet}} integrations across all namespaces](/reference/fleet/data-streams-scenario1.md) shows how to create and use the `logs@custom` and `metrics@custom` component templates to customize {{ilm-init}} policies associated with data streams in integrations.
:::

## Scenario [example-using-index-lifecycle-policy-scenario]
Expand Down Expand Up @@ -140,7 +144,7 @@ Copies of managed {{ilm-init}} policies are also marked as **Managed**. You can

To apply your new {{ilm-init}} policy to the `logs` index template, create or edit the `logs@custom` component template.

A `*@custom` component template allows you to customize the mappings and settings of managed index templates, without having to override managed index templates or component templates. This type of component template is automatically picked up by the index template. [Learn more](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template).
A `*@custom` component template allows you to customize the mappings and settings of managed index templates, without having to override managed index templates or component templates. This type of component template is automatically picked up by the index template.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sentence with *@custom component template feels a bit weird.

Apparently it implies that any random *@custom component template will be magically picked up by an index template. So if I create mydata@custom it will be picked up by a template.

I think what we want to mean here is that we can create logs@custom, metrics@custom, traces@custom, and others (but not any), and those will be picked because they belong to the default index templates created by the stack, right?

Maybe we can convert this list into a table and include for each ILM policy the associated stack index template that uses it?

image

And then in this section we can explain that we can create any *@custom component template where * is any of the default index templates, but not any random string....

Copy link
Contributor

@eedugon eedugon Oct 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, maybe an easier solution just with a text suggestion :) Let me know your thoughts.

Suggested change
A `*@custom` component template allows you to customize the mappings and settings of managed index templates, without having to override managed index templates or component templates. This type of component template is automatically picked up by the index template.
A `<name-of-managed-index-template>@custom` component template allows you to customize the mappings and settings of the managed index templates, without having to override them or their main component templates. This type of component template, which doesn't exist by default, is already included in all managed index templates to allow customizations.

Not sure if my goal is clear, if you understand it and you think there's a better way please go ahead!

Copy link
Contributor Author

@kilfoyle kilfoyle Oct 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eedugon, thanks for this! I think this is such a difficult concept so I tried to really clear it up. What do you think of this?

By the way, I didn't use <name-of-managed-index-template>@custom because (as I just learned) many managed index templates have more than one associated custom component template. Also, I've found that only the main managed index templates have a @custom component template associated with them. So complicated! :-)

screen

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!
Probably we need to include hints about @Custom component templates in more places.
Great explanation and solution! Thanks for digging into this.


:::{tip}
If you want your {{ilm-init}} changes to apply only to specific indices, you can create a custom index template directly instead of modifying the custom component template. Use the **Index management** page in {{kib}} or the [index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API to create a new template.
Expand Down