diff --git a/manage-data/data-store/data-streams/modify-data-stream.md b/manage-data/data-store/data-streams/modify-data-stream.md index e77793f0b0..850ff3f807 100644 --- a/manage-data/data-store/data-streams/modify-data-stream.md +++ b/manage-data/data-store/data-streams/modify-data-stream.md @@ -214,11 +214,10 @@ To change a [dynamic index setting](elasticsearch://reference/elasticsearch/inde ::::{important} -To change the `index.lifecycle.name` setting, first use the [remove policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-remove-policy) to remove the existing {{ilm-init}} policy. See [Switch lifecycle policies](../../lifecycle/index-lifecycle-management/configure-lifecycle-policy.md#switch-lifecycle-policies). +To change the `index.lifecycle.name` setting, first use the [remove policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-remove-policy) to remove the existing {{ilm-init}} policy. See [Switch lifecycle policies](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md#switch-lifecycle-policies). :::: - ### Change a static index setting for a data stream [change-static-index-setting-for-a-data-stream] [Static index settings](elasticsearch://reference/elasticsearch/index-settings/index.md) can only be set when a backing index is created. You cannot update static index settings using the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). diff --git a/manage-data/images/elasticsearch-reference-create-policy-downsample.png b/manage-data/images/elasticsearch-reference-create-policy-downsample.png new file mode 100644 index 0000000000..dfebb1b7fe Binary files /dev/null and b/manage-data/images/elasticsearch-reference-create-policy-downsample.png differ diff --git a/manage-data/images/elasticsearch-reference-create-policy.png b/manage-data/images/elasticsearch-reference-create-policy.png index 738069bf1a..1a1a77312a 100644 Binary files a/manage-data/images/elasticsearch-reference-create-policy.png and b/manage-data/images/elasticsearch-reference-create-policy.png differ diff --git a/manage-data/images/elasticsearch-reference-create-template-wizard-my_template.png b/manage-data/images/elasticsearch-reference-create-template-wizard-my_template.png index fd712e2a6e..e6ab813a33 100644 Binary files a/manage-data/images/elasticsearch-reference-create-template-wizard-my_template.png and b/manage-data/images/elasticsearch-reference-create-template-wizard-my_template.png differ diff --git a/manage-data/images/elasticsearch-reference-datastream-status.png b/manage-data/images/elasticsearch-reference-datastream-status.png new file mode 100644 index 0000000000..56d4032feb Binary files /dev/null and b/manage-data/images/elasticsearch-reference-datastream-status.png differ diff --git a/manage-data/images/elasticsearch-reference-ilm-status.png b/manage-data/images/elasticsearch-reference-ilm-status.png new file mode 100644 index 0000000000..4aa8befbfb Binary files /dev/null and b/manage-data/images/elasticsearch-reference-ilm-status.png differ diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index c8b001d69f..c888608896 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -390,7 +390,7 @@ When data reaches the `cold` or `frozen` phases, it is automatically converted t 5. Restore indices from the searchable snapshots. 1. Follow the steps to [specify the data tier based allocation inclusion rules](/manage-data/lifecycle/data-tiers.md#update-data-tier-allocation-rules). - 2. Remove the associated ILM policy (set it to `null`). If you want to apply a different ILM policy, follow the steps to [Switch lifecycle policies](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md#switch-lifecycle-policies). + 2. Remove the associated ILM policy (set it to `null`). If you want to apply a different ILM policy, follow the steps to [Switch lifecycle policies](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md#switch-lifecycle-policies). 3. If needed, specify the alias for rollover, otherwise set it to `null`. 4. Optionally, specify the desired number of replica shards. diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index c465a6f7e0..a7e26d08ef 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -68,12 +68,13 @@ For example, if you are indexing metrics data from a fleet of ATMs into Elastics You can create and manage index lifecycle policies through {{kib}}'s [Index Management](/manage-data/data-store/index-basics.md#index-management) UI or the [{{ilm-init}} APIs](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-ilm). For more details on creating and managing index lifecycle policies refer to: * [Configure a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) -* [Update a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md) +* [View the lifecycle status of an index](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md) +* [Update or switch a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md) * [Start and stop index lifecycle management](/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md) * [Restore a managed data stream or index](/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md) * [Customize built-in policies](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) -Default {{ilm}} policies are created automatically when you use {{agent}}, {{beats}}, or the {{ls}} {{es}} output plugin to send data to the {{stack}}. +Default {{ilm}} policies are created automatically when you install an [Elastic Integration](integration-docs://reference/index.md), or when you use {{agent}}, {{beats}}, or the {{ls}} {{es}} output plugin to send data to the {{stack}}. ![index lifecycle policies](/manage-data/images/elasticsearch-reference-index-lifecycle-policies.png) diff --git a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md index 39c05d4da0..1ba2e37a67 100644 --- a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md +++ b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md @@ -10,30 +10,77 @@ products: # Configure a lifecycle policy [set-up-lifecycle-policy] -For {{ilm-init}} to manage an index, a valid policy must be specified in the `index.lifecycle.name` index setting. +An [{{ilm}}](/manage-data/lifecycle/index-lifecycle-management.md) ({{ilm-init}}) policy defines how your indices are managed over time, automating when and how they transition as they age. -To configure a lifecycle policy for [rolling indices](rollover.md), you create the policy and add it to the [index template](../../data-store/templates.md). +**Consider these aspects when configuring an {{ilm-init}} policy:** -To use a policy to manage an index that doesn’t roll over, you can specify a lifecycle policy when you create the index, or apply a policy directly to an existing index. +* For {{ilm-init}} to manage an index, you need to specify a valid policy in the `index.lifecycle.name` index setting. -{{ilm-init}} policies are stored in the global cluster state and can be included in snapshots by setting `include_global_state` to `true` when you [take the snapshot](../../../deploy-manage/tools/snapshot-and-restore/create-snapshots.md). When the snapshot is restored, all of the policies in the global state are restored and any local policies with the same names are overwritten. +* To configure a lifecycle policy for [rolling indices](rollover.md) or data streams, you create the policy and add it to the [index template](../../data-store/templates.md). -::::{important} -When you enable {{ilm}} for {{beats}} or the {{ls}} {{es}} output plugin, the necessary policies and configuration changes are applied automatically. You can modify the default policies, but you do not need to explicitly configure a policy or bootstrap an initial index. -:::: +* To use a policy to manage a single index, you can specify a lifecycle policy when you create the index, or apply a policy directly to an existing index. + +* {{ilm-init}} policies are stored in the global cluster state and can be included in snapshots by setting `include_global_state` to `true` when you [take the snapshot](../../../deploy-manage/tools/snapshot-and-restore/create-snapshots.md). When the snapshot is restored, all of the policies in the global state are restored and any local policies with the same names are overwritten. + +:::{note} +This page is specifically about {{ilm-init}}. If you're looking for a simpler lifecycle management option for data streams, refer to [Data stream lifecycle](/manage-data/lifecycle/data-stream.md). Check [Data lifecycle](/manage-data/lifecycle.md) to compare these lifecycle management options. +::: + +## Overview [ilm-configure-overview] + +To set up ILM to manage one or more indices, the general procedure is as follows: + +1. [Create a lifecycle policy](#ilm-create-policy) +2. [Create an index template to apply the lifecycle policy](#apply-policy-template) +If you're configuring ILM for rolling indices and not using [data streams](../../data-store/data-streams.md), you additionally need to: -## Create lifecycle policy [ilm-create-policy] +3. [Create an initial managed index](#create-initial-index) -To create a lifecycle policy from {{kib}}, open the menu and go to **Stack Management > Index Lifecycle Policies**. Click **Create policy**. +You can perform these actions in {{kib}} or using the {{es}} API. -![Create policy page](/manage-data/images/elasticsearch-reference-create-policy.png "") +## Create a lifecycle policy [ilm-create-policy] -You specify the lifecycle phases for the policy and the actions to perform in each phase. +A lifecycle policy defines a set of index lifecycle phases and the actions to perform on the managed indices in each phase. -The [create or update policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-put-lifecycle) API is invoked to add the policy to the {{es}} cluster. +::::{tab-set} +:group: kibana-api +:::{tab-item} {{kib}} +:sync: kibana +To add an ILM policy to an {{es}} cluster: + +1. Go to **Stack Management > Index Lifecycle Policies**, and select **Create policy**. + + ![Create policy page](/manage-data/images/elasticsearch-reference-create-policy.png "") + +1. Specify a name for the lifecycle policy. Later on, when you create an index template to define how indices are created, you'll use this name to assign the lifecycle policy to each index. + +1. In the **Hot phase**, by default an ILM-managed index [rolls over](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) when either: + * It reaches 30 days of age. + * One or more primary shards reach 50 GB in size. + + Disable **Use recommended defaults** to adjust these values or to roll over based on the size of the primary shard, the number of documents in the primary shard, or the total number of documents in the index. + + ::::{important} +The rollover action implicitly rolls over a data stream or alias if one or more shards contain 200,000,000 or more documents. Typically, a shard will reach 50GB before it reaches 200M documents, however, this isn’t the case for space efficient data sets. This built-in limit exists to avoid Search performance loss if a shard contains more than 200M documents. + :::: + +1. By default, only the hot index lifecycle phase is enabled. Enable each additional lifecycle phase that you'd like, and for each choose any [index lifecycle actions](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md) to perform on indices when they enter that phase. + + For example, you could choose the action to [downsample](/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) the index, which aggregates the time series data into statistical summaries, reducing the data storage size. + + ![Create policy page](/manage-data/images/elasticsearch-reference-create-policy-downsample.png "") + + Note that for each phase after the hot phase, you have the option to move the data into the next phase after a certain duration of time. This duration is calculated from the time of the index rollover and not from the time the index is created. + + +1. For the final phase that's enabled, choose to either keep the data in the phase forever or delete the data after a specified period of time. +::: + +:::{tab-item} API +:sync: api +Use the [Create or update policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-put-lifecycle) API to add an ILM policy to the {{es}} cluster: -::::{dropdown} API example ```console PUT _ilm/policy/my_policy { @@ -57,35 +104,84 @@ PUT _ilm/policy/my_policy } ``` -1. Roll over the index when it reaches 25GB in size +1. Roll over the index when one or more primary shards reach 25GB in size. 2. Delete the index 30 days after rollover - +::::{important} +The rollover action implicitly rolls over a data stream or alias if one or more shards contain 200,000,000 or more documents. Typically, a shard will reach 50GB before it reaches 200M documents, however, this isn’t the case for space efficient data sets. This built-in limit exists to avoid Search performance loss if a shard contains more than 200M documents. +::: :::: +## Create an index template to apply the lifecycle policy [apply-policy-template] -::::{important} -The rollover action implicitly always rolls over a data stream or alias if one or more shards contain 200000000 or more documents. Normally a shard will reach 25GB long before it reaches 200M documents, but this isn’t the case for space efficient data sets. Search performance will very likely suffer if a shard contains more than 200M documents. This is the reason of the builtin limit. -:::: +To use a lifecycle policy that triggers a rollover action, you need to configure the policy in the index template used to create each new index. You specify the name of the policy and the alias used to reference the rolling indices. +:::{tip} +If you already have an index template to which you'd like to add an {{ilm-init}} policy, you can do this from **Stack Management > Index Lifecycle Policies**. Search for and select the policy you want, and from the **Actions** menu, select **Add to index template**. +::: +::::{tab-set} +:group: kibana-api +:::{tab-item} {{kib}} +:sync: kibana +To add an index template to a cluster and apply the lifecycle policy to indices matching the template: -## Apply lifecycle policy with an index template [apply-policy-template] +1. Go to **Stack Management > Index Management**. In the **Index Templates** tab, select **Create template**. -To use a policy that triggers the rollover action, you need to configure the policy in the index template used to create each new index. You specify the name of the policy and the alias used to reference the rolling indices. + ![Create template page](/manage-data/images/elasticsearch-reference-create-template-wizard-my_template.png "") -::::{tip} -An `index.lifecycle.rollover_alias` setting is only required if using {{ilm}} with an alias. It is unnecessary when using [Data Streams](../../data-store/data-streams.md). -:::: +1. On the **Logistics** page: + 1. Specify a name for the template. + 1. Specify a pattern to match the indices you want to manage with the lifecycle policy. For example, `my-index-*`. + 1. If you're storing continuously generated, append-only data, you can opt to create [data streams](/manage-data/data-store/data-streams.md) instead of indices for more efficient storage. + :::{note} + When you enable the data stream option, an option to set **Data retention** also becomes available. Since you're creating an index lifecycle policy to manage indices, the **Data retention** option must remain disabled. Data retention is applicable only if you're using a data stream lifecycle, which is an alternative to ILM. Refer to the [Data stream lifecycle](/manage-data/lifecycle/data-stream.md) to learn more. + ::: -You can use the {{kib}} Create template wizard to create a template. To access the wizard, open the menu and go to **Stack Management > Index Management**. In the **Index Templates** tab, click **Create template**. -![Create template page](/manage-data/images/elasticsearch-reference-create-template-wizard-my_template.png "") + 1. Configure any other options you'd like, including: + * The [index mode](elasticsearch://reference/elasticsearch/index-settings/time-series.md) to use for the created indices. + * The template priority, version, and any metadata. + * Whether or not to overwrite the `action.auto_create_index` cluster setting. + + Refer to the [Create or update index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) documentation for details about these options. -The wizard invokes the [create or update index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) to add templates to a cluster. +1. On the **Component templates** page, use the search and filter tools to select any [component templates](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) to include in the index template. The index template will inherit the settings, mappings, and aliases defined in the component templates and apply them to indices when they're created. + +1. On the **Index settings** page: + 1. Configure ILM by specifying the [ILM settings](https://www.elastic.co/docs/api/doc/elasticsearch/configuration-reference/index-lifecycle-management-settings#_index_level_settings_2) to apply to the indices: + * `index.lifecycle.name` - The lifecycle policy to manage the created indices. + * `index.lifecycle.rollover_alias` - The index [alias](/manage-data/data-store/aliases.md) used for querying and managing the set of indices associated with a lifecycle policy that contains a rollover action. + + :::{tip} + The `index.lifecycle.rollover_alias` setting is required only if you're using {{ilm}} with an alias. It is unnecessary when using [Data Streams](../../data-store/data-streams.md). + ::: + + 1. Optional: Add any additional [index settings](elasticsearch://reference/elasticsearch/index-settings/index.md), that should be applied to the indices as they're created. For example, you can set the number of shards and replicas for each index: + + ```json + { + "index.lifecycle.name": "my_policy", + "index.lifecycle.rollover_alias": "test-alias", + "number_of_shards": 1, + "number_of_replicas": 1 + } + ``` + +1. Optional: On the **Mappings** page, customize the fields and data types used when documents are indexed into {{es}}. Refer to [Mapping](/manage-data/data-store/mapping.md) for details. + +1. Optional: On the **Aliases** page, specify an [alias](/manage-data/data-store/aliases.md) for each created index. This isn't required when configuring ILM, which instead uses the `index.lifecycle.rollover_alias` setting to acceess rolling indices. + +1. On the **Review** page, confirm your selections. You can check your selected options, as well as both the format of the index template that will be created and the associated API request. + +The newly created index template will be used for all new indices with names that match the specified pattern, and for each of these, the specified ILM policy will be applied. +::: + +:::{tab-item} API +:sync: api +Use the [Create or update index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) to add an index template to a cluster and apply the lifecycle policy to indices matching the template: -::::{dropdown} API example ```console PUT _index_template/my_template { @@ -101,141 +197,75 @@ PUT _index_template/my_template } ``` -1. Use this template for all new indices whose names begin with `test-` -2. Apply `my_policy` to new indices created with this template -3. Define an index alias for referencing indices managed by `my_policy` - +1. Use this template for all new indices with names that begin with `test-`. +2. Apply `my_policy` to new indices created with this template. +3. Define an index alias for referencing indices managed by `my_policy`. + :::{tip} + The `index.lifecycle.rollover_alias` setting is required only if you're using {{ilm}} with an alias. It is unnecessary when using [Data Streams](../../data-store/data-streams.md). + ::: +::: :::: +## Create an initial managed index [create-initial-index] +When you set up policies for your own rolling indices and are not using the recommended [data streams](../../data-store/data-streams.md), you must manually create the first index managed by a policy and designate it as the write index. -### Create an initial managed index [create-initial-index] - -When you set up policies for your own rolling indices, if you are not using the recommended [data streams](../../data-store/data-streams.md), you need to manually create the first index managed by a policy and designate it as the write index. +The name of the index must match the pattern defined in the index template and end with a number. This number is incremented to generate the name of indices created by the rollover action. ::::{important} -When you enable {{ilm}} for {{beats}} or the {{ls}} {{es}} output plugin, the necessary policies and configuration changes are applied automatically. You can modify the default policies, but you do not need to explicitly configure a policy or bootstrap an initial index. +When you enable {{ilm}} for {{beats}}, {{agent}}, or for the {{agent}} or {{ls}} {{es}} output plugins, the necessary policies and configuration changes are applied automatically. If you'd like to create a specialized ILM policy for any data stream, refer to our tutorial [Customize built-in policies](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md). :::: +::::{tab-set} +:group: kibana-api +:::{tab-item} {{kib}} +:sync: kibana +To create the initial managed index: -The name of the index must match the pattern defined in the index template and end with a number. This number is incremented to generate the name of indices created by the rollover action. +1. Go to **Stack Management > Index Management**. In the **Indices** tab, select **Create index**. +1. Specify a name for the index that matches the index template pattern and that ends with a number. For example, `test-000001`. +1. Leave the **Index mode** set to the default **Standard**. -For example, the following request creates the `test-00001` index. Because it matches the index pattern specified in `my_template`, {{es}} automatically applies the settings from that template. +Create an alias for the index: -```console -PUT test-000001 -{ - "aliases": { - "test-alias":{ - "is_write_index": true <1> - } - } -} -``` +1. Open **Dev tools**. +2. Send the following request: -1. Set this initial index to be the write index for this alias. - - -Now you can start indexing data to the rollover alias specified in the lifecycle policy. With the sample `my_policy` policy, the rollover action is triggered once the initial index exceeds 25GB. {{ilm-init}} then creates a new index that becomes the write index for the `test-alias`. - - -## Apply lifecycle policy manually [apply-policy-manually] - -You can specify a policy when you create an index or apply a policy to an existing index through {{kib}} Management or the [update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). When you apply a policy, {{ilm-init}} immediately starts managing the index. - -::::{important} -Do not manually apply a policy that uses the rollover action. Policies that use rollover must be applied by the [index template](#apply-policy-template). Otherwise, the policy is not carried forward when the rollover action creates a new index. -:::: - - -The `index.lifecycle.name` setting specifies an index’s policy. - -::::{dropdown} API example ```console -PUT test-index +POST /_aliases { - "settings": { - "number_of_shards": 1, - "number_of_replicas": 1, - "index.lifecycle.name": "my_policy" <1> - } + "actions" : [ + { "add" : { "index" : "my-index", "alias" : "my-alias" } } <1> + ] } ``` -1. Sets the lifecycle policy for the index. - - -:::: - +1. Replace `my-index` with the name of the initial managed index that you created previously and set `my-alias` to the rollover alias specified by `index.lifecycle.rollover_alias` in the index template. +Now you can start indexing data to the rollover alias specified in the lifecycle policy. With the sample `my_policy` policy, the rollover action is triggered once the initial index exceeds 50GB. {{ilm-init}} then creates a new index that becomes the write index for the `test-alias`. +::: -### Apply a policy to multiple indices [apply-policy-multiple] - -You can apply the same policy to multiple indices by using wildcards in the index name when you call the [update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) API. - -::::{warning} -Be careful that you don’t inadvertently match indices that you don’t want to modify. -:::: +:::{tab-item} API +:sync: api +Use the [Create an index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) to create the initial managed index. +The following request creates the `test-000001` index. Because it matches the index pattern specified in `my_template`, {{es}} automatically applies the settings from that template. ```console -PUT mylogs-pre-ilm*/_settings <1> +PUT test-000001 { - "index": { - "lifecycle": { - "name": "mylogs_policy_existing" + "aliases": { + "test-alias":{ + "is_write_index": true <1> } } } ``` -1. Updates all indices with names that start with `mylogs-pre-ilm` - - - -### Switch lifecycle policies [switch-lifecycle-policies] - -To switch an index’s lifecycle policy, follow these steps: - -1. Remove the existing policy using the [remove policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-remove-policy). Target a data stream or alias to remove the policies of all its indices. - - ```console - POST logs-my_app-default/_ilm/remove - ``` - -2. The remove policy API removes all {{ilm-init}} metadata from the index and doesn’t consider the index’s lifecycle status. This can leave indices in an undesired state. - - For example, the [`forcemerge`](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) action temporarily closes an index before reopening it. Removing an index’s {{ilm-init}} policy during a `forcemerge` can leave the index closed indefinitely. - - After policy removal, use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to check an index’s state . Target a data stream or alias to get the state of all its indices. - - ```console - GET logs-my_app-default - ``` - - You can then change the index as needed. For example, you can re-open any closed indices using the [open index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-open). - - ```console - POST logs-my_app-default/_open - ``` - -3. Assign a new policy using the [update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). Target a data stream or alias to assign a policy to all its indices. - - ::::{warning} - Don’t assign a new policy without first removing the existing policy. This can cause [phase execution](index-lifecycle.md#ilm-phase-execution) to silently fail. - :::: - - - ```console - PUT logs-my_app-default/_settings - { - "index": { - "lifecycle": { - "name": "new-lifecycle-policy" - } - } - } - ``` +1. Set this initial index to be the write index for this alias. +Now you can start indexing data to the rollover alias specified in the lifecycle policy. With the sample `my_policy` policy, the rollover action is triggered once the initial index exceeds 25GB. {{ilm-init}} then creates a new index that becomes the write index for the `test-alias`. +::: +:::: diff --git a/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md b/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md index 8419f82a70..154146b6a8 100644 --- a/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md +++ b/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md @@ -22,7 +22,7 @@ Starting in Curator version 5.7, Curator ignores {{ilm-init}} managed indices. ## Apply policies to existing time series indices [ilm-existing-indices-apply] -The simplest way to transition to managing your periodic indices with {{ilm-init}} is to [configure an index template](configure-lifecycle-policy.md#apply-policy-template) to apply a lifecycle policy to new indices. Once the index you are writing to is being managed by {{ilm-init}}, you can [manually apply a policy](configure-lifecycle-policy.md#apply-policy-multiple) to your older indices. +The simplest way to transition to managing your periodic indices with {{ilm-init}} is to [configure an index template](configure-lifecycle-policy.md#apply-policy-template) to apply a lifecycle policy to new indices. Once the index you are writing to is being managed by {{ilm-init}}, you can [manually apply a policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md) to your older indices. Define a separate policy for your older indices that omits the rollover action. Rollover is used to manage where new data goes, so isn’t applicable. diff --git a/manage-data/lifecycle/index-lifecycle-management/policy-apply.md b/manage-data/lifecycle/index-lifecycle-management/policy-apply.md new file mode 100644 index 0000000000..55351b76f7 --- /dev/null +++ b/manage-data/lifecycle/index-lifecycle-management/policy-apply.md @@ -0,0 +1,71 @@ +--- +navigation_title: Manually apply a policy to an index +applies_to: + stack: ga +products: + - id: elasticsearch +--- + +# Manually apply a lifecycle policy to an index [apply-policy-manually] + +When you create new {{es}} index you can use an index template to apply the lifecycle policy by which the index will be managed. This process is described in [Configure a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md). + +You can also manually apply a lifecycle policy to an existing index, as described on this page. If an index is currently managed by an ILM policy you must first remove that policy before applying a new one. Refer to [Switch to a different lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md#switch-lifecycle-policies) for details. + +You can do this procedure in {{kib}} or using the {{es}} API. + +::::{warning} +Do not manually apply a policy that uses the rollover action. Policies that use rollover must be applied by the [index template](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md#apply-policy-template). Otherwise, the policy is not carried forward when the rollover action creates a new index. +:::: + +::::{tab-set} +:group: kibana-api +:::{tab-item} {{kib}} +:sync: kibana + +To apply a lifecycle policy to an existing index: + +1. Go to **Stack Management > Index Management**. In the **Indices** tab, search for and select an index. Enable **Include hidden indices** to see the full list of indices. + +1. From the **Manage index** dropdown menu select **Add lifecycle policy**. + +1. Choose a lifecycle policy and confirm your changes. + +::: + +:::{tab-item} API +:sync: api +Use the [update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) to apply a lifecycle policy to an index. + +```console +PUT my-index/_settings +{ + "index": { + "lifecycle": { + "name": "my_ilm_policy" + } + } +} +``` + +You can apply the same policy to multiple indices by using wildcards in the index name when you call the API. + +```console +PUT my-indices*/_settings +{ + "index": { + "lifecycle": { + "name": "my_ilm_policy" + } + } +} +``` + +::::::{warning} +Be careful not to inadvertently match indices that you don’t want to modify. +:::::: + +Once the policy is applied, {{ilm-init}} starts managing the index immediately. + +::: +:::: diff --git a/manage-data/lifecycle/index-lifecycle-management/policy-updates.md b/manage-data/lifecycle/index-lifecycle-management/policy-updates.md index 9afcb5c0e8..34682da919 100644 --- a/manage-data/lifecycle/index-lifecycle-management/policy-updates.md +++ b/manage-data/lifecycle/index-lifecycle-management/policy-updates.md @@ -1,5 +1,5 @@ --- -navigation_title: Update a policy +navigation_title: Update or switch a policy mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/update-lifecycle-policy.html applies_to: @@ -8,7 +8,7 @@ products: - id: elasticsearch --- -# Update a lifecycle policy +# Update or switch a lifecycle policy You can change how the lifecycle of an index or collection of rolling indices is managed by modifying the current policy or switching to a different policy. @@ -31,3 +31,133 @@ For example, if you create a policy that has a hot phase that does not specify a When you apply a different policy to a managed index, the index completes the current phase using the cached definition from the previous policy. The index starts using the new policy when it moves to the next phase. + +## Update an existing lifecycle policy [update-lifecycle-policy] + +You can update a lifecycle policy that is currently associated with one or more indices. + +:::{warning} +Avoid changing any managed policies that are shipped with {{es}}, such as `logs@lifecycle` or `metrics@lifecycle`. Instead, create a new, [custom ILM policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) and associate it with the intended index template or indices. +::: + +::::{tab-set} +:group: kibana-api +:::{tab-item} {{kib}} +:sync: kibana +To update a lifecycle policy: + +1. Go to **Stack Management > Index Lifecycle Policies** and use the search tool to find the lifecycle policy that you want to update. + + You can deselect the **Include managed system policies** option to filter out managed policies from the list, since it's strongly recommended not to update these. + +1. Check the links in the **Linked index templates** and **Linked indices** columns to confirm that your updates will apply only to templates or indices that you want to affect with a new policy. + +1. For the policy that you want to update, select the `edit` icon in the **Actions** menu. + + Note that from the **Actions** menu you can also choose to add the ILM policy to any existing index templates. + +1. On the **Edit policy** page, enable any {{ilm-init}} phases as needed, and expand **Advanced settings** to adjust the [index lifecycle actions](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md) configured for that phase. +::: + + + +:::{tab-item} API +:sync: api +Use the [Create or update policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-put-lifecycle) API to update an existing ILM policy: + +```console +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover": { + "max_primary_shard_size": "25GB" <1> + } + } + }, + "delete": { + "min_age": "30d", + "actions": { + "delete": {} <2> + } + } + } + } +} +``` + +1. Roll over the index when one or more of its shards reach 25GB in size. +2. Delete the index 30 days after rollover. + + +The specified policy will be replaced and the policy version is incremented. +::: +:::: + + +## Switch to a different lifecycle policy [switch-lifecycle-policies] + +You can change an index to be managed by a different {{ilm-init}} policy. + +:::::{warning} +When you remove an ILM policy, all {{ilm-init}} metadata is removed from the managed index without consideration of the index’s lifecycle status. This can leave indices in an undesired state. + +For example, in certain cases the [`forcemerge`](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) action temporarily closes an index before reopening it. Removing an index’s {{ilm-init}} policy during a `forcemerge` can leave the index closed until it is manually reopened. +::::: + +::::{tab-set} +:group: kibana-api +:::{tab-item} {{kib}} +:sync: kibana +To switch an index’s lifecycle policy: + +1. Go to **Stack Management > Index Management**. In the **Indices** tab, search for and select the index that you that you want to switch to a new policy. You can use the **Lifecycle status** filter to narrow the list. + +1. From the **Manage index** dropdown menu, select **Remove lifecycle policy**. Confirm your choice before the ILM policy is removed. + +1. From the **Manage index** dropdown menu, select **Add lifecycle policy**, and then select a new policy to apply. +::: + +:::{tab-item} API +:sync: api +Use the {{es}} [remove policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-remove-policy) and [update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) APIs to switch an index’s lifecycle policy: + +1. Remove the existing policy using the [remove policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-remove-policy). Target a data stream or alias to remove the policies of all its indices. + + ```console + POST logs-my_app-default/_ilm/remove + ``` + +2. Use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to check an index’s state. Target a data stream or alias to get the state of all its indices. + + ```console + GET logs-my_app-default + ``` + + You can then change the index as needed. For example, you can re-open any closed indices using the [open index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-open). + + ```console + POST logs-my_app-default/_open + ``` + +3. Assign a new policy using the [update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). Target a data stream or alias to assign a policy to all its indices. + + ::::::{warning} + * Don’t assign a new policy without first removing the existing policy. This can cause [phase execution](index-lifecycle.md#ilm-phase-execution) to silently fail. + :::::: + + ```console + PUT logs-my_app-default/_settings + { + "index": { + "lifecycle": { + "name": "new-lifecycle-policy" + } + } + } + ``` + +::: +:::: \ No newline at end of file diff --git a/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md b/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md new file mode 100644 index 0000000000..539932db27 --- /dev/null +++ b/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md @@ -0,0 +1,138 @@ +--- +navigation_title: View the lifecycle status of an index +mapped_pages: + - https://www.elastic.co/guide/en/elasticsearch/reference/current/update-lifecycle-policy.html +applies_to: + stack: ga +products: + - id: elasticsearch +--- + +# View the lifecycle status of an index [view-lifecycle-status] + +For any existing managed index in your cluster, you can access the ILM policy applied to it and details such as its current phase. + +:::::{tab-set} +:group: kibana-api +::::{tab-item} {{kib}} +:sync: kibana +**To view the current lifecycle status for one or more indices:** + +1. Go to **Stack Management > Index Management** and open the **Indices** tab. +1. Enable **Include hidden indices** to view all indices, including those managed by ILM. Note that if you're using data streams, you can find the data stream associated with any index listed in the **Data stream** column. +1. Use the search tool to find the index you're looking for. You can also filter the results by lifecycle status and lifecycle phase. +1. Select the index to view details. +1. Open the **Index lifecycle** tab to view ILM details such as the current lifecycle phase, the ILM policy name, the current [index lifecycle action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md), and other details. + + ![Index lifecycle status page](/manage-data/images/elasticsearch-reference-ilm-status.png "") + +:::{tip} +{{es}} comes with many built-in ILM policies. For standard Observability or Security use cases, you will have two {{ilm-init}} policies configured automatically: `logs@lifecycle` for logs and `metrics@lifecycle` for metrics. + +To learn how to create a specialized ILM policy for any data stream, such as those created when you install an Elastic Integration, refer to our tutorial [Customize built-in policies](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md). +::: + +**To view the current lifecycle status for a datastream:** + +1. Go to **Stack Management > Index Management** and open the **Data Streams** tab. +1. Use the search tool to find the data stream you're looking for. +1. Select the data stream to view details. The flyout that opens includes direct links to the ILM policy and the index template. + + ![Data stream status page](/manage-data/images/elasticsearch-reference-datastream-status.png "") +:::: + +:::{tab-item} API +:sync: api +Use the [Explain the lifecycle state API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-explain-lifecycle) to view the current lifecycle status for an index: + +```console +GET .ds-metrics-system.process-default-2025.06.04-000001/_ilm/explain +``` + +Tthe API response shows the current ILM phase and other details: + +```json +{ + "indices": { + ".ds-metrics-system.process-default-2025.06.04-000001": { + "index": ".ds-metrics-system.process-default-2025.06.04-000001", + "managed": true, + "policy": "metrics", + "index_creation_date_millis": 1749060710358, + "time_since_index_creation": "22.91d", + "lifecycle_date_millis": 1749060710358, + "age": "22.91d", + "phase": "hot", + "phase_time_millis": 1749060711038, + "action": "rollover", + "action_time_millis": 1749060712038, + "step": "check-rollover-ready", + "step_time_millis": 1749060712038, + "phase_execution": { + "policy": "metrics", + "phase_definition": { + "min_age": "0ms", + "actions": { + "rollover": { + "max_age": "30d", + "max_primary_shard_docs": 200000000, + "min_docs": 1, + "max_primary_shard_size": "50gb" + } + } + }, + "version": 1, + "modified_date_in_millis": 1749059754363 + } + } + } +} +``` + +You can also call this API for a data stream: + +```console +GET metrics-system.process-default/_ilm/explain +``` + +When called for a data stream, the API retrieves the current lifecycle status for the stream's backing indices: + +```json +{ + "indices": { + ".ds-metrics-system.process-default-2025.06.04-000001": { + "index": ".ds-metrics-system.process-default-2025.06.04-000001", + "managed": true, + "policy": "metrics", + "index_creation_date_millis": 1749060710358, + "time_since_index_creation": "22.91d", + "lifecycle_date_millis": 1749060710358, + "age": "22.91d", + "phase": "hot", + "phase_time_millis": 1749060711038, + "action": "rollover", + "action_time_millis": 1749060712038, + "step": "check-rollover-ready", + "step_time_millis": 1749060712038, + "phase_execution": { + "policy": "metrics", + "phase_definition": { + "min_age": "0ms", + "actions": { + "rollover": { + "max_age": "30d", + "max_primary_shard_docs": 200000000, + "min_docs": 1, + "max_primary_shard_size": "50gb" + } + } + }, + "version": 1, + "modified_date_in_millis": 1749059754363 + } + } + } +} +``` +:::: +::::: \ No newline at end of file diff --git a/manage-data/toc.yml b/manage-data/toc.yml index 41e0cd82d9..bbca9ac4a0 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -122,6 +122,8 @@ toc: - file: lifecycle/index-lifecycle-management/skip-rollover.md - file: lifecycle/index-lifecycle-management/tutorial-automate-rollover.md - file: lifecycle/index-lifecycle-management/configure-lifecycle-policy.md + - file: lifecycle/index-lifecycle-management/policy-apply.md + - file: lifecycle/index-lifecycle-management/policy-view-status.md - file: lifecycle/index-lifecycle-management/policy-updates.md - file: lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md - file: lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md