From d0d630313613c341dc325a0b7d47ed61ffda8344 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 14 Feb 2025 10:20:31 -0600 Subject: [PATCH 1/5] first draft of ilm intro --- .../lifecycle/index-lifecycle-management.md | 79 ++++++++++++++++--- 1 file changed, 69 insertions(+), 10 deletions(-) diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index 8e8ea13bc9..98f45a71aa 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -2,15 +2,12 @@ mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html - https://www.elastic.co/guide/en/elasticsearch/reference/current/overview-index-lifecycle-management.html - - https://www.elastic.co/guide/en/serverless/current/elasticsearch-differences.html#elasticsearch-differences-serverless-features-replaced - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrate-index-management.html - https://www.elastic.co/guide/en/cloud/current/ec-configure-index-management.html - https://www.elastic.co/guide/en/cloud/current/ec-migrate-index-management.html - - https://www.elastic.co/guide/en/cloud/current/ec-disable-data-tier.html + - https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-concepts.html --- -# Index lifecycle management - % What needs to be done: Refine % GitHub issue: docs-projects#376 @@ -19,13 +16,12 @@ mapped_urls: % Use migrated content from existing pages that map to this page: -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-lifecycle-management.md -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md -% - [ ] ./raw-migrated-files/docs-content/serverless/elasticsearch-differences.md -% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-migrate-index-management.md +% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-lifecycle-management.md +% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md +% - [x] ./raw-migrated-files/docs-content/serverless/elasticsearch-differences.md % - [ ] ./raw-migrated-files/cloud/cloud/ec-configure-index-management.md +% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-migrate-index-management.md % - [ ] ./raw-migrated-files/cloud/cloud/ec-migrate-index-management.md -% - [ ] ./raw-migrated-files/cloud/cloud/ec-disable-data-tier.md % Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): @@ -45,4 +41,67 @@ $$$ilm-gs-alias-apply-policy$$$ $$$ilm-gs-alias-check-progress$$$ -$$$ec-update-data-tier-allocation-rules$$$ \ No newline at end of file +# Index lifecycle management + +{{ilm-cap}} ({{ilm-init}}) provides an integrated and streamlined way to manage time-based data such as logs and metrics, making it easier to follow best practices for managing your indices. + +You can configure {{ilm-init}} policies to automatically manage indices according to your performance, resiliency, and retention requirements. For example, you could use {{ilm-init}} to: + +* Spin up a new index when an index reaches a certain size or number of documents +* Create a new index each day, week, or month and archive previous ones +* Delete stale indices to enforce data retention standards + +::::{tip} +{{ilm-init}} is not available on {{es-serverless}}. + +:::{dropdown} Why? +In an {{ecloud}} or self-managed environment, ILM lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {{es-serverless}} eliminates this complexity by optimizing your cluster performance for you. + +Data stream lifecycle is an optimized lifecycle tool that lets you focus on the most common lifecycle management needs, without unnecessary hardware-centric concepts like data tiers. +::: +:::: + +::::{important} +To use {{ilm-init}}, all nodes in a cluster must run the same version. Although it might be possible to create and apply policies in a mixed-version cluster, there is no guarantee they will work as intended. Attempting to use a policy that contains actions that aren’t supported on all nodes in a cluster will cause errors. +:::: + + +## Trigger actions + +{{ilm-init}} policies can trigger actions like: + +* **Rollover**: Creates a new write index when the current one reaches a certain size, number of docs, or age. +* **Shrink**: Reduces the number of primary shards in an index. +* **Force merge**: Triggers a [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html) to reduce the number of segments in an index’s shards. +* **Delete**: Permanently remove an index, including all of its data and metadata. +* [And more](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-actions.html) + +Each action has options you can use to specify index behavior and characteristics like: + +* The maximum shard size, number of documents, or age at which you want to roll over to a new index. +* The point at which the index is no longer being updated and the number of primary shards can be reduced. +* When to force a merge to permanently remove documents marked for deletion. +* The point at which the index can be moved to less performant hardware. +* The point at which the availability is not as critical and the number of replicas can be reduced. +* When the index can be safely deleted. + +For example, if you are indexing metrics data from a fleet of ATMs into Elasticsearch, you might define a policy that says: + +1. When the total size of the index’s primary shards reaches 50GB, roll over to a new index. +2. Move the old index into the warm phase, mark it read only, and shrink it down to a single shard. +3. After 7 days, move the index into the cold phase and move it to less expensive hardware. +4. Delete the index once the required 30 day retention period is reached. + +**Learn about all available actions in [Index lifecycle actions](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-actions.html).** + +## Create and manage {{ilm-init}} policies + +You can create and manage index lifecycle policies through [{{kib}} Management](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) or the [{{ilm-init}} APIs](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-ilm). + +Default {{ilm}} policies are created automatically when you use {{agent}}, {{beats}}, or the {{ls}} {{es}} output plugin to send data to the {{stack}}. + +![index lifecycle policies](../../images/elasticsearch-reference-index-lifecycle-policies.png) + +::::{tip} +To automatically back up your indices and manage snapshots, use [snapshot lifecycle policies](/deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm). +:::: \ No newline at end of file From 24f59caaf1facb3d8d88cfd7a71a67616097e6da Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 14 Feb 2025 10:21:15 -0600 Subject: [PATCH 2/5] reorder ilm toc --- .../index-lifecycle-management/concepts.md | 18 ---------------- .../configure-lifecycle-policy.md | 2 +- .../index-management-in-kibana.md | 2 +- .../policy-updates.md | 10 ++++----- .../start-stop-index-lifecycle-management.md | 9 ++++---- .../tutorial-automate-rollover.md | 6 ------ .../tutorial-customize-built-in-policies.md | 6 +----- manage-data/toc.yml | 21 +++++++++---------- 8 files changed, 22 insertions(+), 52 deletions(-) delete mode 100644 manage-data/lifecycle/index-lifecycle-management/concepts.md diff --git a/manage-data/lifecycle/index-lifecycle-management/concepts.md b/manage-data/lifecycle/index-lifecycle-management/concepts.md deleted file mode 100644 index ec1637001e..0000000000 --- a/manage-data/lifecycle/index-lifecycle-management/concepts.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -navigation_title: "Concepts" -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-concepts.html ---- - - - -# Concepts [ilm-concepts] - - -* [Index lifecycle](index-lifecycle.md) -* [Rollover](rollover.md) -* [Policy updates](policy-updates.md) - - - - diff --git a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md index 09117f1e6b..12fa569009 100644 --- a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md +++ b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md @@ -1,4 +1,5 @@ --- +navigation_title: Configure a policy mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html --- @@ -18,7 +19,6 @@ When you enable {{ilm}} for {{beats}} or the {{ls}} {es} output plugin, the nece :::: - ## Create lifecycle policy [ilm-create-policy] To create a lifecycle policy from {{kib}}, open the menu and go to **Stack Management > Index Lifecycle Policies**. Click **Create policy**. diff --git a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md index 203f07cfbf..25b3c08b86 100644 --- a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md +++ b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md @@ -12,7 +12,7 @@ To use these features, go to **Stack Management** > **Index Management**. ## Required permissions [index-mgm-req-permissions] -If you use {{es}} {security-features}, the following [security privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) are required: +If you use {{es}} {{security-features}}, the following [security privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) are required: * The `monitor` cluster privilege to access {{kib}}'s **Index Management** features. * The `view_index_metadata` and `manage` index privileges to view a data stream or index’s data. diff --git a/manage-data/lifecycle/index-lifecycle-management/policy-updates.md b/manage-data/lifecycle/index-lifecycle-management/policy-updates.md index 8ffb1906bd..e19df18461 100644 --- a/manage-data/lifecycle/index-lifecycle-management/policy-updates.md +++ b/manage-data/lifecycle/index-lifecycle-management/policy-updates.md @@ -1,12 +1,10 @@ --- -navigation_title: "Policy updates" +navigation_title: "Update a policy" mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/update-lifecycle-policy.html --- - - -# Policy updates [update-lifecycle-policy] +# Update a lifecycle policy You can change how the lifecycle of an index or collection of rolling indices is managed by modifying the current policy or switching to a different policy. @@ -16,7 +14,7 @@ To ensure that policy updates don’t put an index into a state where it can’t When the index advances to the next phase, it uses the phase definition from the updated policy. -## How changes are applied [ilm-apply-changes] +## How changes are applied [ilm-apply-changes] When a policy is initially applied to an index, the index gets the latest version of the policy. If you update the policy, the policy version is bumped and {{ilm-init}} can detect that the index is using an earlier version that needs to be updated. @@ -25,7 +23,7 @@ Changes to `min_age` are not propagated to the cached definition. Changing a pha For example, if you create a policy that has a hot phase that does not specify a `min_age`, indices immediately enter the hot phase when the policy is applied. If you then update the policy to specify a `min_age` of 1 day for the hot phase, that has no effect on indices that are already in the hot phase. Indices created *after* the policy update won’t enter the hot phase until they are a day old. -## How new policies are applied [ilm-apply-new-policy] +## How new policies are applied [ilm-apply-new-policy] When you apply a different policy to a managed index, the index completes the current phase using the cached definition from the previous policy. The index starts using the new policy when it moves to the next phase. diff --git a/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md index 9c9bbdc858..806eb8ba0f 100644 --- a/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md @@ -1,4 +1,5 @@ --- +navigation_title: Start and stop {{ilm-init}} mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/start-stop-ilm.html --- @@ -9,13 +10,13 @@ By default, the {{ilm-init}} service is in the `RUNNING` state and manages all i You can stop {{ilm}} to suspend management operations for all indices. For example, you might stop {{ilm}} when performing scheduled maintenance or making changes to the cluster that could impact the execution of {{ilm-init}} actions. -::::{important} +::::{important} When you stop {{ilm-init}}, [{{slm-init}}](../../../deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm) operations are also suspended. No snapshots will be taken as scheduled until you restart {{ilm-init}}. In-progress snapshots are not affected. :::: -## Get {{ilm-init}} status [get-ilm-status] +## Get {{ilm-init}} status [get-ilm-status] To see the current status of the {{ilm-init}} service, use the [Get Status API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-get-status.html): @@ -32,7 +33,7 @@ Under normal operation, the response shows {{ilm-init}} is `RUNNING`: ``` -## Stop {{ilm-init}} [stop-ilm] +## Stop {{ilm-init}} [stop-ilm] To stop the {{ilm-init}} service and pause execution of all lifecycle policies, use the [Stop API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-stop.html): @@ -57,7 +58,7 @@ Once all policies are at a safe stopping point, {{ilm-init}} moves into the `STO ``` -## Start {{ilm-init}} [_start_ilm_init] +## Start {{ilm-init}} [_start_ilm_init] To restart {{ilm-init}} and resume executing policies, use the [Start API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-start.html). This puts the {{ilm-init}} service in the `RUNNING` state and {{ilm-init}} begins executing policies from where it left off. diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md index 33756623cd..6b0225fa63 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md @@ -1,22 +1,16 @@ --- -navigation_title: "Tutorial" mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html --- - - # Tutorial: Automate rollover [getting-started-index-lifecycle-management] - When you continuously index timestamped documents into {{es}}, you typically use a [data stream](../../data-store/index-types/data-streams.md) so you can periodically [roll over](rollover.md) to a new index. This enables you to implement a [hot-warm-cold architecture](../data-tiers.md) to meet your performance requirements for your newest data, control costs over time, enforce retention policies, and still get the most out of your data. ::::{tip} [Data streams](../../data-store/index-types/data-streams.md) are best suited for [append-only](../../data-store/index-types/data-streams.md#data-streams-append-only) use cases. If you need to update or delete existing time series data, you can perform update or delete operations directly on the data stream backing index. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. You can still use [ILM](../index-lifecycle-management.md) to manage and [roll over](rollover.md) the alias’s indices. Skip to [Manage time series data without data streams](../index-lifecycle-management.md#manage-time-series-data-without-data-streams). :::: - - ## Manage time series data with data streams [manage-time-series-data-with-data-streams] To automate rollover and management of a data stream with {{ilm-init}}, you: diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md index 349790545f..cf4f7399f3 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md @@ -1,13 +1,9 @@ --- -navigation_title: "Tutorial" mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/example-using-index-lifecycle-policy.html --- - - -# Tutorial: Customize built-in policies [example-using-index-lifecycle-policy] - +# Tutorial: Customize built-in policies {{es}} includes the following built-in {{ilm-init}} policies: diff --git a/manage-data/toc.yml b/manage-data/toc.yml index 91b422474b..5a14e2296d 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -98,7 +98,7 @@ toc: - file: ingest/sample-data.md - file: ingest/transform-enrich.md children: - - file: ingest/transform-enrich/ingest-pipelines-serverless.md + - file: ingest/transform-enrich/ingest-pipelines-serverless.md - file: ingest/transform-enrich/ingest-pipelines.md children: - file: ingest/transform-enrich/example-parse-logs.md @@ -118,20 +118,19 @@ toc: - file: lifecycle/data-tiers.md - file: lifecycle/index-lifecycle-management.md children: - - file: lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md - - file: lifecycle/index-lifecycle-management/tutorial-automate-rollover.md - - file: lifecycle/index-lifecycle-management/index-management-in-kibana.md - - file: lifecycle/index-lifecycle-management/concepts.md + - file: lifecycle/index-lifecycle-management/index-lifecycle.md + - file: lifecycle/index-lifecycle-management/rollover.md children: - - file: lifecycle/index-lifecycle-management/index-lifecycle.md - - file: lifecycle/index-lifecycle-management/rollover.md - - file: lifecycle/index-lifecycle-management/policy-updates.md + - file: lifecycle/index-lifecycle-management/skip-rollover.md + - file: lifecycle/index-lifecycle-management/tutorial-automate-rollover.md - file: lifecycle/index-lifecycle-management/configure-lifecycle-policy.md - - file: lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md + - file: lifecycle/index-lifecycle-management/policy-updates.md - file: lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md - - file: lifecycle/index-lifecycle-management/manage-existing-indices.md - - file: lifecycle/index-lifecycle-management/skip-rollover.md - file: lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md + - file: lifecycle/index-lifecycle-management/index-management-in-kibana.md + - file: lifecycle/index-lifecycle-management/manage-existing-indices.md + - file: lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md + - file: lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md - file: lifecycle/data-stream.md children: - file: lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md From 8790a79a7303e594195fbd6e647cf0d2035fdd62 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 14 Feb 2025 12:29:44 -0600 Subject: [PATCH 3/5] clean up --- .../lifecycle/index-lifecycle-management.md | 37 +++++------ .../migrate-index-management.md | 63 +++++++++++++++++++ .../tutorial-automate-rollover.md | 18 +++--- manage-data/toc.yml | 3 +- .../cloud/ec-configure-index-management.md | 17 ----- .../cloud/ec-migrate-index-management.md | 19 ++++-- .../elasticsearch-reference/data-streams.md | 4 +- .../index-lifecycle-management.md | 31 --------- .../overview-index-lifecycle-management.md | 38 ----------- raw-migrated-files/toc.yml | 3 - .../index-lifecycle-management-errors.md | 2 +- 11 files changed, 107 insertions(+), 128 deletions(-) create mode 100644 manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md delete mode 100644 raw-migrated-files/cloud/cloud/ec-configure-index-management.md delete mode 100644 raw-migrated-files/elasticsearch/elasticsearch-reference/index-lifecycle-management.md delete mode 100644 raw-migrated-files/elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index 97a9e596c9..26410d1b5d 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -19,28 +19,12 @@ mapped_urls: % - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-lifecycle-management.md % - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md % - [x] ./raw-migrated-files/docs-content/serverless/elasticsearch-differences.md -% - [ ] ./raw-migrated-files/cloud/cloud/ec-configure-index-management.md +% - [x] ./raw-migrated-files/cloud/cloud/ec-configure-index-management.md % - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-migrate-index-management.md % - [ ] ./raw-migrated-files/cloud/cloud/ec-migrate-index-management.md % Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): -$$$ilm-gs-alias-bootstrap$$$ - -$$$manage-time-series-data-without-data-streams$$$ - -$$$ilm-gs-create-policy$$$ - -$$$ilm-gs-apply-policy$$$ - -$$$ilm-gs-create-the-data-stream$$$ - -$$$ilm-gs-check-progress$$$ - -$$$ilm-gs-alias-apply-policy$$$ - -$$$ilm-gs-alias-check-progress$$$ - # Index lifecycle management {{ilm-cap}} ({{ilm-init}}) provides an integrated and streamlined way to manage time-based data such as logs and metrics, making it easier to follow best practices for managing your indices. @@ -65,8 +49,7 @@ Data stream lifecycle is an optimized lifecycle tool that lets you focus on the To use {{ilm-init}}, all nodes in a cluster must run the same version. Although it might be possible to create and apply policies in a mixed-version cluster, there is no guarantee they will work as intended. Attempting to use a policy that contains actions that aren’t supported on all nodes in a cluster will cause errors. :::: - -## Trigger actions +## Actions {{ilm-init}} policies can trigger actions like: @@ -96,7 +79,13 @@ For example, if you are indexing metrics data from a fleet of ATMs into Elastics ## Create and manage {{ilm-init}} policies -You can create and manage index lifecycle policies through [{{kib}} Management](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) or the [{{ilm-init}} APIs](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-ilm). +You can create and manage index lifecycle policies through [{{kib}} Management](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) or the [{{ilm-init}} APIs](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-ilm). For more details on creating and managing index lifecycle policies refer to: + +* [Configure a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) +* [Update a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md) +* [Start and stop index lifecycle management](/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md) +* [Restore a managed data stream or index](/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md) +* [Customize built-in policies](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) Default {{ilm}} policies are created automatically when you use {{agent}}, {{beats}}, or the {{ls}} {{es}} output plugin to send data to the {{stack}}. @@ -105,3 +94,11 @@ Default {{ilm}} policies are created automatically when you use {{agent}}, {{bea ::::{tip} To automatically back up your indices and manage snapshots, use [snapshot lifecycle policies](/deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm). :::: + +## Migrate to {{ilm-init}} + +For existing hot-warm deployments that are currently using index curation, migrating to ILM gives you more fine-grained control over the lifecycle of each index. Read more in: + +* [Manage existing indices](/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md) +* [Migrate to index lifecycle management](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md) +* [Migrate index allocation filters to node roles](/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md) diff --git a/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md b/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md new file mode 100644 index 0000000000..cf11ae4bb2 --- /dev/null +++ b/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md @@ -0,0 +1,63 @@ +--- +mapped_pages: + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrate-index-management.html + - https://www.elastic.co/guide/en/cloud/current/ec-configure-index-management.html + - https://www.elastic.co/guide/en/cloud/current/ec-migrate-index-management.html +--- + +# Migrate to index lifecycle management + +::::{important} +Index curation is deprecated. Any deployments using index curation are prompted to migrate to ILM. +:::: + +The index lifecycle management (ILM) feature of the {{stack}} provides an integrated and streamlined way to manage time-based data, making it easier to follow best practices for managing your indices. Compared to index curation, migrating to ILM gives you more fine-grained control over the lifecycle of each index. + +For existing hot-warm deployments that are currently using index curation, there are a couple of options for migrating to index lifecycle management (ILM). You can: + +* Use the migration process in the console to change an existing deployment to ILM. +* Take a snapshot and restore your data to a new Elastic Stack deployment that has ILM enabled. + +To configure ILM Migration in the console: + +::::{tab-set} + +:::{tab-item} {{ech}} +1. Log in to the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body). +2. From the **Deployments** page, select your deployment. + + On the deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. + +3. Near the top of the deployment overview, you should get a message to migrate from index curation to index lifecycle management (ILM) along with a **Start migration** button. +4. Select which index curation pattern you wish to migrate. +5. Set the ILM policy name for each data view (formerly *index pattern*). +::: + +:::{tab-item} Elastic Cloud Enterprise +1. [Log into the Cloud UI](../../../deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). +2. From the **Deployments** page, select your deployment. + + Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. + +3. Near the top of the deployment overview, you should get a message to migrate from index curation to index lifecycle management (ILM) along with a **Start migration** button. +4. Select which index curation pattern you wish to migrate. +5. Set the ILM policy name for each data view (formerly *index pattern*). +::: + +:::: + +::::{note} +Beginning with Elastic Stack version 8.0, Kibana *index patterns* have been renamed to *data views*. To learn more, check the Kibana [What’s new in 8.0](https://www.elastic.co/guide/en/kibana/8.0/whats-new.html#index-pattern-rename) page. +:::: + +1. Set the shard allocation attribute for the data view. + + * You can set different node attributes per data view to allow for more in-depth configuration in Kibana, or + * You may choose to add one node attribute that applies to all data views. + * If you do not wish to migrate a certain data view to ILM, you can deselect the checkbox in the associated row. + * You may also wish to migrate to ILM without carrying over any of your current data views by deselecting all patterns. This means that those data views will no longer be curated, and you will have the option to set up new ILM policies in Kibana. + +2. Select **Migrate**. + +After you get the notification that confirms that migration was completed successfully, you can view your ILM policies in Kibana. + diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md index e7d2f0475c..e5c62b7ed0 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md @@ -8,16 +8,16 @@ mapped_pages: When you continuously index timestamped documents into {{es}}, you typically use a [data stream](../../data-store/index-types/data-streams.md) so you can periodically [roll over](rollover.md) to a new index. This enables you to implement a [hot-warm-cold architecture](../data-tiers.md) to meet your performance requirements for your newest data, control costs over time, enforce retention policies, and still get the most out of your data. ::::{tip} -[Data streams](../../data-store/index-types/data-streams.md) are best suited for [append-only](../../data-store/index-types/data-streams.md#data-streams-append-only) use cases. If you need to update or delete existing time series data, you can perform update or delete operations directly on the data stream backing index. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. You can still use [ILM](../index-lifecycle-management.md) to manage and [roll over](rollover.md) the alias’s indices. Skip to [Manage time series data without data streams](../index-lifecycle-management.md#manage-time-series-data-without-data-streams). +[Data streams](../../data-store/index-types/data-streams.md) are best suited for [append-only](../../data-store/index-types/data-streams.md#data-streams-append-only) use cases. If you need to update or delete existing time series data, you can perform update or delete operations directly on the data stream backing index. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. You can still use [ILM](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md) to manage and [roll over](rollover.md) the alias’s indices. Skip to [Manage time series data without data streams](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams). :::: ## Manage time series data with data streams [manage-time-series-data-with-data-streams] To automate rollover and management of a data stream with {{ilm-init}}, you: -1. [Create a lifecycle policy](../index-lifecycle-management.md#ilm-gs-create-policy) that defines the appropriate [phases](index-lifecycle.md) and [actions](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-actions.html). -2. [Create an index template](../index-lifecycle-management.md#ilm-gs-apply-policy) to [create the data stream](../index-lifecycle-management.md#ilm-gs-create-the-data-stream) and apply the ILM policy and the indices settings and mappings configurations for the backing indices. -3. [Verify indices are moving through the lifecycle phases](../index-lifecycle-management.md#ilm-gs-check-progress) as expected. +1. [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-create-policy) that defines the appropriate [phases](index-lifecycle.md) and [actions](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-actions.html). +2. [Create an index template](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-apply-policy) to [create the data stream](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-create-the-data-stream) and apply the ILM policy and the indices settings and mappings configurations for the backing indices. +3. [Verify indices are moving through the lifecycle phases](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-check-progress) as expected. ::::{important} When you enable {{ilm}} for {{beats}} or the {{ls}} {{es}} output plugin, lifecycle policies are set up automatically. You do not need to take any other actions. You can modify the default policies through [{{kib}} Management](tutorial-customize-built-in-policies.md) or the {{ilm-init}} APIs. @@ -214,10 +214,10 @@ If you frequently send multiple documents using the same `_id` expecting last-wr To automate rollover and management of time series indices with {{ilm-init}} using an index alias, you: -1. Create a lifecycle policy that defines the appropriate phases and actions. See [Create a lifecycle policy](../index-lifecycle-management.md#ilm-gs-create-policy) above. -2. [Create an index template](../index-lifecycle-management.md#ilm-gs-alias-apply-policy) to apply the policy to each new index. -3. [Bootstrap an index](../index-lifecycle-management.md#ilm-gs-alias-bootstrap) as the initial write index. -4. [Verify indices are moving through the lifecycle phases](../index-lifecycle-management.md#ilm-gs-alias-check-progress) as expected. +1. Create a lifecycle policy that defines the appropriate phases and actions. See [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-create-policy) above. +2. [Create an index template](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-alias-apply-policy) to apply the policy to each new index. +3. [Bootstrap an index](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-alias-bootstrap) as the initial write index. +4. [Verify indices are moving through the lifecycle phases](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-alias-check-progress) as expected. ### Create an index template to apply the lifecycle policy [ilm-gs-alias-apply-policy] @@ -285,7 +285,7 @@ This process repeats each time rollover conditions are met. You can search acros ### Check lifecycle progress [ilm-gs-alias-check-progress] -Retrieving the status information for managed indices is very similar to the data stream case. See the data stream [check progress section](../index-lifecycle-management.md#ilm-gs-check-progress) for more information. The only difference is the indices namespace, so retrieving the progress will entail the following api call: +Retrieving the status information for managed indices is very similar to the data stream case. See the data stream [check progress section](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-check-progress) for more information. The only difference is the indices namespace, so retrieving the progress will entail the following api call: ```console GET timeseries-*/_ilm/explain diff --git a/manage-data/toc.yml b/manage-data/toc.yml index 15063fb543..e0223ef3b6 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -127,10 +127,11 @@ toc: - file: lifecycle/index-lifecycle-management/policy-updates.md - file: lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md - file: lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md + - file: lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md - file: lifecycle/index-lifecycle-management/index-management-in-kibana.md - file: lifecycle/index-lifecycle-management/manage-existing-indices.md + - file: lifecycle/index-lifecycle-management/migrate-index-management.md - file: lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md - - file: lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md - file: lifecycle/data-stream.md children: - file: lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md diff --git a/raw-migrated-files/cloud/cloud/ec-configure-index-management.md b/raw-migrated-files/cloud/cloud/ec-configure-index-management.md deleted file mode 100644 index e0eb7d4aad..0000000000 --- a/raw-migrated-files/cloud/cloud/ec-configure-index-management.md +++ /dev/null @@ -1,17 +0,0 @@ -# Configure index management [ec-configure-index-management] - -::::{important} -Index curation is deprecated. Any deployments using index curation will be prompted to migrate to ILM. -:::: - - -The index lifecycle management (ILM) feature of the Elastic Stack provides an integrated and streamlined way to manage time-based data, making it easier to follow best practices for managing your indices. Compared to index curation, migrating to ILM gives you more fine-grained control over the lifecycle of each index. - -For existing hot-warm deployments that are currently using index curation, there are a couple of options for migrating to index lifecycle management (ILM). You can: - -* Use the migration process in the console to change an existing deployment to ILM. -* Take a snapshot and restore your data to a new Elastic Stack 6.7+ deployment that has ILM enabled. -* [Create your index lifecyle policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html) -* [Managing the index lifecycle](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) - - diff --git a/raw-migrated-files/cloud/cloud/ec-migrate-index-management.md b/raw-migrated-files/cloud/cloud/ec-migrate-index-management.md index ea7d82e60e..8c226a0b6a 100644 --- a/raw-migrated-files/cloud/cloud/ec-migrate-index-management.md +++ b/raw-migrated-files/cloud/cloud/ec-migrate-index-management.md @@ -12,13 +12,11 @@ For existing hot-warm deployments that are currently using index curation, there * Use the migration process in the console to change an existing deployment to ILM. * Take a snapshot and restore your data to a new Elastic Stack deployment that has ILM enabled. -To learn more about configuring index lifecycle management for Elasticsearch Service or about all of the features that are available with ILM, see: - -* [Create your index lifecyle policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html) -* [Managing the index lifecycle](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) - To configure ILM Migration in the console: +::::{tab-set} + +:::{tab-item} {{ech}} 1. Log in to the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body). 2. From the **Deployments** page, select your deployment. @@ -27,12 +25,21 @@ To configure ILM Migration in the console: 3. Near the top of the deployment overview, you should get a message to migrate from index curation to index lifecycle management (ILM) along with a **Start migration** button. 4. Select which index curation pattern you wish to migrate. 5. Set the ILM policy name for each data view (formerly *index pattern*). +::: + +:::{tab-item} Elastic Cloud Enterprise +1. [Log into the Cloud UI](../../../deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). +2. From the **Deployments** page, select your deployment. + + Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. +::: + +:::: ::::{note} Beginning with Elastic Stack version 8.0, Kibana *index patterns* have been renamed to *data views*. To learn more, check the Kibana [What’s new in 8.0](https://www.elastic.co/guide/en/kibana/8.0/whats-new.html#index-pattern-rename) page. :::: - 1. Set the shard allocation attribute for the data view. * You can set different node attributes per data view to allow for more in-depth configuration in Kibana, or diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/data-streams.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/data-streams.md index befa980ff7..f648c339ff 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/data-streams.md +++ b/raw-migrated-files/elasticsearch/elasticsearch-reference/data-streams.md @@ -18,7 +18,7 @@ To determine whether you should use a data stream for your data, you should cons * You mostly perform indexing requests, with occasional updates and deletes. * You index documents without an `_id`, or when indexing documents with an explicit `_id` you expect first-write-wins behavior. -For most time series data use-cases, a data stream will be a good fit. However, if you find that your data doesn’t fit into these categories (for example, if you frequently send multiple documents using the same `_id` expecting last-write-wins), you may want to use an index alias with a write index instead. See documentation for [managing time series data without a data stream](../../../manage-data/lifecycle/index-lifecycle-management.md#manage-time-series-data-without-data-streams) for more information. +For most time series data use-cases, a data stream will be a good fit. However, if you find that your data doesn’t fit into these categories (for example, if you frequently send multiple documents using the same `_id` expecting last-write-wins), you may want to use an index alias with a write index instead. See documentation for [managing time series data without a data stream](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams) for more information. Keep in mind that some features such as [Time Series Data Streams (TSDS)](../../../manage-data/data-store/index-types/tsdb.md) and [data stream lifecycles](../../../manage-data/lifecycle/data-stream.md) require a data stream. @@ -98,7 +98,7 @@ Data streams are designed for use cases where existing data is rarely updated. Y If you need to update a larger number of documents in a data stream, you can use the [update by query](../../../manage-data/data-store/index-types/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](../../../manage-data/data-store/index-types/use-data-stream.md#delete-docs-in-a-data-stream-by-query) APIs. ::::{tip} -If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. See [Manage time series data without data streams](../../../manage-data/lifecycle/index-lifecycle-management.md#manage-time-series-data-without-data-streams). +If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. See [Manage time series data without data streams](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams). :::: diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/index-lifecycle-management.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/index-lifecycle-management.md deleted file mode 100644 index 066e1c305c..0000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/index-lifecycle-management.md +++ /dev/null @@ -1,31 +0,0 @@ -# {{ilm-init}}: Manage the index lifecycle [index-lifecycle-management] - -You can configure {{ilm}} ({{ilm-init}}) policies to automatically manage indices according to your performance, resiliency, and retention requirements. For example, you could use {{ilm-init}} to: - -* Spin up a new index when an index reaches a certain size or number of documents -* Create a new index each day, week, or month and archive previous ones -* Delete stale indices to enforce data retention standards - -You can create and manage index lifecycle policies through {{kib}} Management or the {{ilm-init}} APIs. Default {{ilm}} policies are created automatically when you use {{agent}}, {{beats}}, or the {{ls}} {{es}} output plugin to send data to the {{stack}}. - -![index lifecycle policies](../../../images/elasticsearch-reference-index-lifecycle-policies.png "") - -::::{tip} -To automatically back up your indices and manage snapshots, use [snapshot lifecycle policies](../../../deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm). -:::: - - -* [Tutorial: Customize built-in policies](../../../manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) -* [Tutorial: Automate rollover](../../../manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md) -* [Overview](../../../manage-data/lifecycle/index-lifecycle-management.md) -* [Concepts](../../../manage-data/lifecycle/index-lifecycle-management/concepts.md) -* [*Configure a lifecycle policy*](../../../manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) -* [*Migrate index allocation filters to node roles*](../../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md) -* [*Troubleshooting {{ilm}} errors*](../../../troubleshoot/elasticsearch/elasticsearch-reference/index-lifecycle-management-errors.md) -* [*Start and stop {{ilm}}*](../../../manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md) -* [*Manage existing indices*](../../../manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md) -* [*Skip rollover*](../../../manage-data/lifecycle/index-lifecycle-management/skip-rollover.md) -* [*Restore a managed data stream or index*](../../../manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md) -* [*{{ilm-cap}} APIs*](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ilm) -* [*Index lifecycle actions*](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-actions.html) - diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md deleted file mode 100644 index 8ca368ac68..0000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -navigation_title: "Overview" ---- - -# {{ilm-init}} overview [overview-index-lifecycle-management] - - -You can create and apply {{ilm-cap}} ({{ilm-init}}) policies to automatically manage your indices according to your performance, resiliency, and retention requirements. - -Index lifecycle policies can trigger actions such as: - -* **Rollover**: Creates a new write index when the current one reaches a certain size, number of docs, or age. -* **Shrink**: Reduces the number of primary shards in an index. -* **Force merge**: Triggers a [force merge](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) to reduce the number of segments in an index’s shards. -* **Delete**: Permanently remove an index, including all of its data and metadata. - -{{ilm-init}} makes it easier to manage indices in hot-warm-cold architectures, which are common when you’re working with time series data such as logs and metrics. - -You can specify: - -* The maximum shard size, number of documents, or age at which you want to roll over to a new index. -* The point at which the index is no longer being updated and the number of primary shards can be reduced. -* When to force a merge to permanently remove documents marked for deletion. -* The point at which the index can be moved to less performant hardware. -* The point at which the availability is not as critical and the number of replicas can be reduced. -* When the index can be safely deleted. - -For example, if you are indexing metrics data from a fleet of ATMs into Elasticsearch, you might define a policy that says: - -1. When the total size of the index’s primary shards reaches 50GB, roll over to a new index. -2. Move the old index into the warm phase, mark it read only, and shrink it down to a single shard. -3. After 7 days, move the index into the cold phase and move it to less expensive hardware. -4. Delete the index once the required 30 day retention period is reached. - -::::{important} -To use {{ilm-init}}, all nodes in a cluster must run the same version. Although it might be possible to create and apply policies in a mixed-version cluster, there is no guarantee they will work as intended. Attempting to use a policy that contains actions that aren’t supported on all nodes in a cluster will cause errors. - -:::: diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index 60b4da22d2..0a48789c41 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -172,7 +172,6 @@ toc: - file: cloud/cloud/ec-billing-stop.md - file: cloud/cloud/ec-cloud-ingest-data.md - file: cloud/cloud/ec-config-change-errors.md - - file: cloud/cloud/ec-configure-index-management.md - file: cloud/cloud/ec-configuring-keystore.md - file: cloud/cloud/ec-custom-bundles.md - file: cloud/cloud/ec-custom-repository.md @@ -537,7 +536,6 @@ toc: - file: elasticsearch/elasticsearch-reference/fips-140-compliance.md - file: elasticsearch/elasticsearch-reference/how-monitoring-works.md - file: elasticsearch/elasticsearch-reference/ignore_missing_component_templates.md - - file: elasticsearch/elasticsearch-reference/index-lifecycle-management.md - file: elasticsearch/elasticsearch-reference/index-mgmt.md - file: elasticsearch/elasticsearch-reference/index-modules-allocation.md - file: elasticsearch/elasticsearch-reference/index-modules-mapper.md @@ -554,7 +552,6 @@ toc: - file: elasticsearch/elasticsearch-reference/native-realm.md - file: elasticsearch/elasticsearch-reference/oidc-guide.md - file: elasticsearch/elasticsearch-reference/oidc-realm.md - - file: elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md - file: elasticsearch/elasticsearch-reference/recovery-prioritization.md - file: elasticsearch/elasticsearch-reference/role-mapping-resources.md - file: elasticsearch/elasticsearch-reference/rollup-overview.md diff --git a/troubleshoot/elasticsearch/elasticsearch-reference/index-lifecycle-management-errors.md b/troubleshoot/elasticsearch/elasticsearch-reference/index-lifecycle-management-errors.md index 2a741cf058..cc2f3eb76b 100644 --- a/troubleshoot/elasticsearch/elasticsearch-reference/index-lifecycle-management-errors.md +++ b/troubleshoot/elasticsearch/elasticsearch-reference/index-lifecycle-management-errors.md @@ -160,7 +160,7 @@ Problems with rollover aliases are a common cause of errors. Consider using [dat ### Rollover alias [x] can point to multiple indices, found duplicated alias [x] in index template [z] [_rollover_alias_x_can_point_to_multiple_indices_found_duplicated_alias_x_in_index_template_z] -The target rollover alias is specified in an index template’s `index.lifecycle.rollover_alias` setting. You need to explicitly configure this alias *one time* when you [bootstrap the initial index](../../../manage-data/lifecycle/index-lifecycle-management.md#ilm-gs-alias-bootstrap). The rollover action then manages setting and updating the alias to [roll over](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover#rollover-index-api-desc) to each subsequent index. +The target rollover alias is specified in an index template’s `index.lifecycle.rollover_alias` setting. You need to explicitly configure this alias *one time* when you [bootstrap the initial index](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-alias-bootstrap). The rollover action then manages setting and updating the alias to [roll over](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover#rollover-index-api-desc) to each subsequent index. Do not explicitly configure this same alias in the aliases section of an index template. From 608ad8d91729a969671f64872839d9eb4e750e11 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 14 Feb 2025 12:42:24 -0600 Subject: [PATCH 4/5] clean up unused files --- .../lifecycle/index-lifecycle-management.md | 6 --- .../ece-migrate-index-management.md | 46 ---------------- .../cloud/ec-migrate-index-management.md | 53 ------------------- raw-migrated-files/toc.yml | 2 - 4 files changed, 107 deletions(-) delete mode 100644 raw-migrated-files/cloud/cloud-enterprise/ece-migrate-index-management.md delete mode 100644 raw-migrated-files/cloud/cloud/ec-migrate-index-management.md diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index 26410d1b5d..2a920cdefd 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -2,9 +2,6 @@ mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html - https://www.elastic.co/guide/en/elasticsearch/reference/current/overview-index-lifecycle-management.html - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrate-index-management.html - - https://www.elastic.co/guide/en/cloud/current/ec-configure-index-management.html - - https://www.elastic.co/guide/en/cloud/current/ec-migrate-index-management.html - https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-concepts.html --- @@ -19,9 +16,6 @@ mapped_urls: % - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-lifecycle-management.md % - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md % - [x] ./raw-migrated-files/docs-content/serverless/elasticsearch-differences.md -% - [x] ./raw-migrated-files/cloud/cloud/ec-configure-index-management.md -% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-migrate-index-management.md -% - [ ] ./raw-migrated-files/cloud/cloud/ec-migrate-index-management.md % Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-migrate-index-management.md b/raw-migrated-files/cloud/cloud-enterprise/ece-migrate-index-management.md deleted file mode 100644 index 78b67d0cec..0000000000 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-migrate-index-management.md +++ /dev/null @@ -1,46 +0,0 @@ -# Migrate to index lifecycle management [ece-migrate-index-management] - -::::{important} -Index curation is deprecated. Any deployments using index curation are prompted to migrate to ILM. -:::: - - -The index lifecycle management (ILM) feature of the {{stack}} provides an integrated and streamlined way to manage time-based data, making it easier to follow best practices for managing your indices. Compared to index curation, migrating to ILM gives you more fine-grained control over the lifecycle of each index. - -For existing hot-warm deployments that are currently using index curation, there are a couple of options for migrating to index lifecycle management (ILM). You can: - -* Use the migration process in the console to change an existing deployment to ILM. -* Take a snapshot and restore your data to a new Elastic Stack deployment that has ILM enabled. - -To learn more about configuring index lifecycle management for Elastic Cloud Enterprise or about all of the features that are available with ILM, see: - -* [Create your index lifecyle policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html) -* [Managing the index lifecycle](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) - -To configure ILM Migration in the console: - -1. [Log into the Cloud UI](../../../deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). -2. From the **Deployments** page, select your deployment. - - Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. - -3. Near the top of the deployment overview, you should get a message to migrate from index curation to index lifecycle management (ILM) along with a **Start migration** button. -4. Select which index curation pattern you wish to migrate. -5. Set the ILM policy name for each data view (formerly *index pattern*). - -::::{note} -Beginning with Elastic Stack version 8.0, Kibana *index patterns* have been renamed to *data views*. To learn more, check the Kibana [What’s new in 8.0](https://www.elastic.co/guide/en/kibana/8.0/whats-new.html#index-pattern-rename) page. -:::: - - -1. Set the shard allocation attribute for the data view. - - * You can set different node attributes per data view to allow for more in-depth configuration in Kibana, or - * You may choose to add one node attribute that applies to all data views. - * If you do not wish to migrate a certain data view to ILM, you can deselect the checkbox in the associated row. - * You may also wish to migrate to ILM without carrying over any of your current data views by deselecting all patterns. This means that those data views will no longer be curated, and you will have the option to set up new ILM policies in Kibana. - -2. Select **Migrate**. - -After you get the notification that confirms that migration was completed successfully, you can view your ILM policies in Kibana. - diff --git a/raw-migrated-files/cloud/cloud/ec-migrate-index-management.md b/raw-migrated-files/cloud/cloud/ec-migrate-index-management.md deleted file mode 100644 index 8c226a0b6a..0000000000 --- a/raw-migrated-files/cloud/cloud/ec-migrate-index-management.md +++ /dev/null @@ -1,53 +0,0 @@ -# Migrate to index lifecycle management [ec-migrate-index-management] - -::::{important} -Index curation is deprecated. Any deployments using index curation are prompted to migrate to ILM. -:::: - - -The index lifecycle management (ILM) feature of the {{stack}} provides an integrated and streamlined way to manage time-based data, making it easier to follow best practices for managing your indices. Compared to index curation, migrating to ILM gives you more fine-grained control over the lifecycle of each index. - -For existing hot-warm deployments that are currently using index curation, there are a couple of options for migrating to index lifecycle management (ILM). You can: - -* Use the migration process in the console to change an existing deployment to ILM. -* Take a snapshot and restore your data to a new Elastic Stack deployment that has ILM enabled. - -To configure ILM Migration in the console: - -::::{tab-set} - -:::{tab-item} {{ech}} -1. Log in to the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. From the **Deployments** page, select your deployment. - - On the deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. - -3. Near the top of the deployment overview, you should get a message to migrate from index curation to index lifecycle management (ILM) along with a **Start migration** button. -4. Select which index curation pattern you wish to migrate. -5. Set the ILM policy name for each data view (formerly *index pattern*). -::: - -:::{tab-item} Elastic Cloud Enterprise -1. [Log into the Cloud UI](../../../deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). -2. From the **Deployments** page, select your deployment. - - Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. -::: - -:::: - -::::{note} -Beginning with Elastic Stack version 8.0, Kibana *index patterns* have been renamed to *data views*. To learn more, check the Kibana [What’s new in 8.0](https://www.elastic.co/guide/en/kibana/8.0/whats-new.html#index-pattern-rename) page. -:::: - -1. Set the shard allocation attribute for the data view. - - * You can set different node attributes per data view to allow for more in-depth configuration in Kibana, or - * You may choose to add one node attribute that applies to all data views. - * If you do not wish to migrate a certain data view to ILM, you can deselect the checkbox in the associated row. - * You may also wish to migrate to ILM without carrying over any of your current data views by deselecting all patterns. This means that those data views will no longer be curated, and you will have the option to set up new ILM policies in Kibana. - -2. Select **Migrate**. - -After you get the notification that confirms that migration was completed successfully, you can view your ILM policies in Kibana. - diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index 0a48789c41..0f9afe13c6 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -73,7 +73,6 @@ toc: - file: cloud/cloud-enterprise/ece-manage-integrations-server.md - file: cloud/cloud-enterprise/ece-manage-kibana-settings.md - file: cloud/cloud-enterprise/ece-manage-kibana.md - - file: cloud/cloud-enterprise/ece-migrate-index-management.md - file: cloud/cloud-enterprise/ece-monitoring-deployments.md - file: cloud/cloud-enterprise/ece-password-reset-elastic.md - file: cloud/cloud-enterprise/ece-restful-api-examples-configuring-keystore.md @@ -200,7 +199,6 @@ toc: - file: cloud/cloud/ec-manage-enterprise-search-settings.md - file: cloud/cloud/ec-manage-kibana-settings.md - file: cloud/cloud/ec-metrics-memory-pressure.md - - file: cloud/cloud/ec-migrate-index-management.md - file: cloud/cloud/ec-monitoring-setup.md - file: cloud/cloud/ec-monitoring.md - file: cloud/cloud/ec-password-reset.md From 63ce05409d52148f882fa4935d9c945fde6f68d8 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Tue, 18 Feb 2025 10:05:34 -0600 Subject: [PATCH 5/5] address feedback, clean up --- .../lifecycle/index-lifecycle-management.md | 14 -------------- .../migrate-index-management.md | 2 +- 2 files changed, 1 insertion(+), 15 deletions(-) diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index 2a920cdefd..9840bad157 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -5,20 +5,6 @@ mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-concepts.html --- -% What needs to be done: Refine - -% GitHub issue: docs-projects#376 - -% Scope notes: Combine the linked resources. The "migrate to ILM" one is probably outdated now. - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-lifecycle-management.md -% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md -% - [x] ./raw-migrated-files/docs-content/serverless/elasticsearch-differences.md - -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): - # Index lifecycle management {{ilm-cap}} ({{ilm-init}}) provides an integrated and streamlined way to manage time-based data such as logs and metrics, making it easier to follow best practices for managing your indices. diff --git a/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md b/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md index cf11ae4bb2..40dca8e06a 100644 --- a/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md +++ b/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md @@ -37,7 +37,7 @@ To configure ILM Migration in the console: 1. [Log into the Cloud UI](../../../deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). 2. From the **Deployments** page, select your deployment. - Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. + On the deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. 3. Near the top of the deployment overview, you should get a message to migrate from index curation to index lifecycle management (ILM) along with a **Start migration** button. 4. Select which index curation pattern you wish to migrate.