Skip to content

Commit a9ebfce

Browse files
committed
Updates for review feedback
1 parent 9ccf2be commit a9ebfce

File tree

7 files changed

+350
-278
lines changed

7 files changed

+350
-278
lines changed
241 KB
Loading
592 KB
Loading

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

Lines changed: 9 additions & 276 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,9 @@ There a few things to note about how an {{ilm-init}} policy works:
1616

1717
* For {{ilm-init}} to manage an index, you need to specify a valid policy in the `index.lifecycle.name` index setting.
1818

19-
* To configure a lifecycle policy for [rolling indices](rollover.md), you create the policy and add it to the [index template](../../data-store/templates.md).
19+
* To configure a lifecycle policy for [rolling indices](rollover.md) or data streams, you create the policy and add it to the [index template](../../data-store/templates.md).
2020

21-
* To use a policy to manage an index that doesn’t roll over, you can specify a lifecycle policy when you create the index, or apply a policy directly to an existing index.
21+
* To use a policy to manage a single index, you can specify a lifecycle policy when you create the index, or apply a policy directly to an existing index.
2222

2323
* {{ilm-init}} policies are stored in the global cluster state and can be included in snapshots by setting `include_global_state` to `true` when you [take the snapshot](../../../deploy-manage/tools/snapshot-and-restore/create-snapshots.md). When the snapshot is restored, all of the policies in the global state are restored and any local policies with the same names are overwritten.
2424

@@ -32,12 +32,6 @@ To set up ILM to manage one or more indices, the general procedure is as follows
3232
If you're configuring ILM for rolling indices and not using [data streams](../../data-store/data-streams.md), you additionally need to:
3333

3434
3. [Create an initial managed index](#create-initial-index)
35-
4. [Apply a lifecycle policy manually](#apply-policy-manually)
36-
37-
Once ILM is set up, at any time you can:
38-
39-
* [View the lifecycle status for an index](#view-lifecycle-status)
40-
* [Switch lifecycle policies](#switch-lifecycle-policies)
4135

4236
You can perform these actions in {{kib}} or using the {{es}} API.
4337

@@ -59,12 +53,12 @@ To add an ILM policy to an {{es}} cluster:
5953

6054
1. In the **Hot phase**, by default an ILM-managed index [rolls over](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) when either:
6155
* It reaches 30 days of age.
62-
* The primary shard reaches 50 gigabytes in size.
56+
* One or more primary shards reach 50 gigabytes in size.
6357

64-
Disaable **Use recommended defaults** to adjust these values or to roll over based on the size of the primary shard, the number of documents in the primary shard, or the total number of documents in the index.
58+
Disable **Use recommended defaults** to adjust these values or to roll over based on the size of the primary shard, the number of documents in the primary shard, or the total number of documents in the index.
6559

6660
::::{important}
67-
The rollover action implicitly always rolls over a data stream or alias if one or more shards contain 200000000 or more documents. Normally a shard will reach 25GB long before it reaches 200M documents, but this isn’t the case for space efficient data sets. Search performance will very likely suffer if a shard contains more than 200M documents. This is the reason for the built-in limit.
61+
The rollover action implicitly always rolls over a data stream or alias if one or more shards contain 200000000 or more documents. Normally a shard will reach 25GB long before it reaches 200M documents, but this isn’t the case for space efficient data sets. Search performance will very likely suffer if a shard contains more than 200M documents. This is the reason for the built-in limit.
6862
::::
6963

7064
1. By default, only the "hot" index lifecycle phase is enabled. Enable each additional lifecycle phase that you'd like, and for each choose the [index lifecycle actions](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md) to perform on indices when they enter that phase.
@@ -129,10 +123,11 @@ To add an index template to a cluster and apply the lifecycle policy to indices
129123
1. Specify a pattern to match the indices you want to manage with the lifecycle policy. For example, `my-index-*`.
130124
1. If you're storing continuously generated, append-only data, you can opt to create [data streams](/manage-data/data-store/data-streams.md) instead of indices for more efficient storage. If you enable this option, you can also enable **Data retention** to configure how long your indexed data is kept.
131125

132-
:::{important}
133-
When the **Data retention** option is set, data is guaranteed to be stored for the specified retention duration. Elasticsearch is allowed at a later time to delete data older than this duration. This setting replaces any data retention settings that may be defined in an ILM policy. Refer to the [Data stream retention](/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md) tutorial to learn more.
126+
:::{note}
127+
Since you're creating an index lifecycle policy to manage indices, the **Data retention** option should be left disabled. Data retention is applicable only if you're using a data stream lifecycle, which is an alternative to ILM. Refer to the [Data stream lifecycle](/manage-data/lifecycle/data-stream.md) to learn more.
134128
:::
135129

130+
136131
1. Configure any other options you'd like, including:
137132
* The [index mode](elasticsearch://reference/elasticsearch/index-settings/time-series.md) to use for the created indices.
138133
* The template priority, version, and any metadata.
@@ -222,7 +217,7 @@ To create the initial managed index:
222217

223218
Create an alias for the index:
224219

225-
1. Open **Developer tools**.
220+
1. Open **Dev tools**.
226221
2. Send the following request:
227222

228223
```console
@@ -262,265 +257,3 @@ Now you can start indexing data to the rollover alias specified in the lifecycle
262257
:::
263258
::::
264259

265-
## Apply a lifecycle policy manually [apply-policy-manually]
266-
267-
You can specify a policy when you create an index or apply a lifecycle policy to an existing index.
268-
269-
::::{important}
270-
Do not manually apply a policy that uses the rollover action. Policies that use rollover must be applied by the [index template](#apply-policy-template). Otherwise, the policy is not carried forward when the rollover action creates a new index.
271-
::::
272-
273-
::::{tab-set}
274-
:group: kibana-api
275-
:::{tab-item} {{kib}}
276-
:sync: kibana
277-
278-
To apply a lifecycle policy to an existing index:
279-
280-
1. Go to **Stack Management > Index Management**. In the **Indices** tab, search for and select the index that you created, for example `test-00001`. Note that to view system indices you need to enable **Include hidden indices**.
281-
282-
1. From the **Manage index** dropdown menu select **Add lifecycle policy**.
283-
284-
1. Choose a lifecycle policy and confirm your changes.
285-
286-
Once the policy is applied, {{ilm-init}} starts managing the index immediately.
287-
288-
% QUESTION: Is there a way in the Kibana UI to add a policy to multiple indices?
289-
:::
290-
291-
:::{tab-item} API
292-
:sync: api
293-
Use the [update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) to apply a lifecycle policy to an existing index.
294-
295-
### Apply an policy to a single index [apply-policy-single]
296-
297-
In the following request, the `index.lifecycle.name` setting specifies an index’s policy.
298-
299-
```console
300-
PUT my-index
301-
{
302-
"settings": {
303-
"number_of_shards": 1,
304-
"number_of_replicas": 1,
305-
"index.lifecycle.name": "my_policy" <1>
306-
}
307-
}
308-
```
309-
310-
1. Sets the lifecycle policy for the index.
311-
312-
Once the policy is applied, {{ilm-init}} starts managing the index immediately.
313-
314-
### Apply a policy to multiple indices [apply-policy-multiple]
315-
316-
You can apply the same policy to multiple indices by using wildcards in the index name when you call the API.
317-
318-
```console
319-
PUT mylogs-pre-ilm*/_settings <1>
320-
{
321-
"index": {
322-
"lifecycle": {
323-
"name": "mylogs_policy_existing"
324-
}
325-
}
326-
}
327-
```
328-
329-
1. Updates all indices with names that start with `mylogs-pre-ilm`.
330-
331-
::::::{warning}
332-
Be careful not to inadvertently match indices that you don’t want to modify.
333-
::::::
334-
:::
335-
::::
336-
337-
## View the lifecycle status for an index [view-lifecycle-status]
338-
339-
For any existing managed index in your cluster, you can access the ILM policy applied to it and details such as its current phase.
340-
341-
::::{tab-set}
342-
:group: kibana-api
343-
:::{tab-item} {{kib}}
344-
:sync: kibana
345-
To view the current lifecycle status for one or more indices:
346-
347-
1. Go to **Stack Management > Index Management** and open the **Indices** tab.
348-
1. Enable **Include hidden indices** to view all indices, including those managed by ILM. Note that if you're using data streams, you can find the data stream associated with any index listed in the **Data stream** column.
349-
1. Use the search tool to find the index you're looking for. You can also filter the results by lifecycle status and lifecycle phase.
350-
1. Select the index to view details.
351-
1. Open the **Index lifecycle** tab to view ILM details such as the current lifecycle phase, the ILM policy name, the current [index lifecycle action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md), and other details.
352-
353-
![Index lifecycle status page](/manage-data/images/elasticsearch-reference-ilm-status.png "")
354-
355-
To view the current lifecycle status for a datastream:
356-
357-
1. Go to **Stack Management > Index Management** and open the **Data Streams** tab.
358-
1. Use the search tool to find the data stream you're looking for.
359-
1. Select the data stream to view details. In the flyout that opens, you have direct links to access the ILM policy and the index template.
360-
361-
![Data stream status page](/manage-data/images/elasticsearch-reference-datastream-status.png "")
362-
:::
363-
364-
:::{tab-item} API
365-
:sync: api
366-
Use the [Explain the lifecycle state API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-explain-lifecycle) to view the current lifecycle status for an index:
367-
368-
```console
369-
GET my-index/_ilm/explain
370-
```
371-
372-
The API response shows the current ILM phase and other details:
373-
374-
```json
375-
{
376-
"indices": {
377-
".ds-logs-elastic_agent.filebeat-default-2025.06.04-000001": {
378-
"index": ".ds-logs-elastic_agent.filebeat-default-2025.06.04-000001",
379-
"managed": true,
380-
"policy": "logs",
381-
"index_creation_date_millis": 1749060710541,
382-
"time_since_index_creation": "19.05d",
383-
"lifecycle_date_millis": 1749060710541,
384-
"age": "19.05d",
385-
"phase": "hot",
386-
"phase_time_millis": 1749060711038,
387-
"action": "rollover",
388-
"action_time_millis": 1749060712038,
389-
"step": "check-rollover-ready",
390-
"step_time_millis": 1749060712038,
391-
"phase_execution": {
392-
"policy": "logs",
393-
"phase_definition": {
394-
"min_age": "0ms",
395-
"actions": {
396-
"rollover": {
397-
"max_age": "30d",
398-
"max_primary_shard_docs": 200000000,
399-
"min_docs": 1,
400-
"max_primary_shard_size": "50gb"
401-
}
402-
}
403-
},
404-
"version": 1,
405-
"modified_date_in_millis": 1749059754275
406-
}
407-
}
408-
}
409-
}
410-
```
411-
412-
You can also call this API for a data stream:
413-
414-
```console
415-
GET my-datastream/_ilm/explain
416-
```
417-
418-
When called for a data stream, the API retrieves the current lifecycle status for the stream's backing indices:
419-
420-
```json
421-
{
422-
"indices": {
423-
".ds-logs-elastic_agent.filebeat-default-2025.06.04-000001": {
424-
"index": ".ds-logs-elastic_agent.filebeat-default-2025.06.04-000001",
425-
"managed": true,
426-
"policy": "logs",
427-
"index_creation_date_millis": 1749060710541,
428-
"time_since_index_creation": "19.06d",
429-
"lifecycle_date_millis": 1749060710541,
430-
"age": "19.06d",
431-
"phase": "hot",
432-
"phase_time_millis": 1749060711038,
433-
"action": "rollover",
434-
"action_time_millis": 1749060712038,
435-
"step": "check-rollover-ready",
436-
"step_time_millis": 1749060712038,
437-
"phase_execution": {
438-
"policy": "logs",
439-
"phase_definition": {
440-
"min_age": "0ms",
441-
"actions": {
442-
"rollover": {
443-
"max_age": "30d",
444-
"max_primary_shard_docs": 200000000,
445-
"min_docs": 1,
446-
"max_primary_shard_size": "50gb"
447-
}
448-
}
449-
},
450-
"version": 1,
451-
"modified_date_in_millis": 1749059754275
452-
}
453-
}
454-
}
455-
}
456-
```
457-
:::
458-
::::
459-
460-
## Switch lifecycle policies [switch-lifecycle-policies]
461-
462-
You may want to change the ILM policy applied to an index. For example, if you're ingesting a large set of log data you may want to apply a different policy to a subset of indices containing only the most critical logs.
463-
464-
:::::{warning}
465-
Be careful when changing either the `logs@lifecycle` or `metrics@lifecycle` policies as these typically manage many indices. In {{kib}}, the **Index Lifecycle Policies** table shows the number of indices currently associated with each policy.
466-
:::::
467-
468-
::::{tab-set}
469-
:group: kibana-api
470-
:::{tab-item} {{kib}}
471-
:sync: kibana
472-
To switch an index’s lifecycle policy:
473-
474-
1. Go to **Stack Management > Index Management**. In the **Indices** tab, search for and select the index that you that you want to switch to a new policy. You can use the **Lifecycle status** filter to narrow the list.
475-
476-
1. From the **Manage index** dropdown menu select **Remove lifecycle policy**. Confirm your choice and then the ILM policy is removed.
477-
478-
1. From the **Manage index** dropdown menu select **Add lifecycle policy**, and then select a new policy to apply.
479-
480-
:::
481-
482-
:::{tab-item} API
483-
:sync: api
484-
Use the {{es}} [remove policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-remove-policy) and [update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) APIs to switch an index’s lifecycle policy:
485-
486-
1. Remove the existing policy using the [remove policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-remove-policy). Target a data stream or alias to remove the policies of all its indices.
487-
488-
```console
489-
POST logs-my_app-default/_ilm/remove
490-
```
491-
492-
2. The remove policy API removes all {{ilm-init}} metadata from the index and doesn’t consider the index’s lifecycle status. This can leave indices in an undesired state.
493-
494-
For example, the [`forcemerge`](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) action temporarily closes an index before reopening it. Removing an index’s {{ilm-init}} policy during a `forcemerge` can leave the index closed indefinitely.
495-
496-
After policy removal, use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to check an index’s state. Target a data stream or alias to get the state of all its indices.
497-
498-
```console
499-
GET logs-my_app-default
500-
```
501-
502-
You can then change the index as needed. For example, you can re-open any closed indices using the [open index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-open).
503-
504-
```console
505-
POST logs-my_app-default/_open
506-
```
507-
508-
3. Assign a new policy using the [update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). Target a data stream or alias to assign a policy to all its indices.
509-
510-
::::::{warning}
511-
* Don’t assign a new policy without first removing the existing policy. This can cause [phase execution](index-lifecycle.md#ilm-phase-execution) to silently fail.
512-
::::::
513-
514-
```console
515-
PUT logs-my_app-default/_settings
516-
{
517-
"index": {
518-
"lifecycle": {
519-
"name": "new-lifecycle-policy"
520-
}
521-
}
522-
}
523-
```
524-
525-
:::
526-
::::

0 commit comments

Comments
 (0)