[manage data] Add ILM intro, clean up ILM section (#458)

Closes elastic/docs-projects#376

Adds an ILM intro page and attempts to reorganize subpages in a logical
order:

* Concepts (Index lifecycle, Rollover)
* Tasks that will be applicable to most users organized roughly
chronologically (Configure a policy, Update a policy, Start and stop
ILM, Restore a managed data stream or index, Customize built-in
policies)
* Index management in Kibana doesn't really fit, but I feel like it
belongs alongside these other pages
* Info on switching to ILM from something else (Manage existing indices,
Migrate to index lifecycle management, Migrate index allocation filters
to node roles)

Author: colleenmcginnis

manage-data/lifecycle/index-lifecycle-management.md

@@ -2,43 +2,83 @@
 mapped_urls:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/overview-index-lifecycle-management.html
-  - https://www.elastic.co/guide/en/serverless/current/elasticsearch-differences.html#elasticsearch-differences-serverless-features-replaced
-  - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrate-index-management.html
-  - https://www.elastic.co/guide/en/cloud/current/ec-configure-index-management.html
-  - https://www.elastic.co/guide/en/cloud/current/ec-migrate-index-management.html
+  - https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-concepts.html
 ---
 
 # Index lifecycle management
 
-% What needs to be done: Refine
+{{ilm-cap}} ({{ilm-init}}) provides an integrated and streamlined way to manage time-based data such as logs and metrics, making it easier to follow best practices for managing your indices.
 
-% GitHub issue: docs-projects#376
+You can configure {{ilm-init}} policies to automatically manage indices according to your performance, resiliency, and retention requirements. For example, you could use {{ilm-init}} to:
 
-% Scope notes: Combine the linked resources. The "migrate to ILM" one is probably outdated now.
+* Spin up a new index when an index reaches a certain size or number of documents
+* Create a new index each day, week, or month and archive previous ones
+* Delete stale indices to enforce data retention standards
 
-% Use migrated content from existing pages that map to this page:
+::::{tip}
+{{ilm-init}} is not available on {{es-serverless}}.
 
-% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-lifecycle-management.md
-% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/overview-index-lifecycle-management.md
-% - [ ] ./raw-migrated-files/docs-content/serverless/elasticsearch-differences.md
-% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-migrate-index-management.md
-% - [ ] ./raw-migrated-files/cloud/cloud/ec-configure-index-management.md
-% - [ ] ./raw-migrated-files/cloud/cloud/ec-migrate-index-management.md
+:::{dropdown} Why?
+In an {{ecloud}} or self-managed environment, ILM lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {{es-serverless}} eliminates this complexity by optimizing your cluster performance for you.
 
-% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):
+Data stream lifecycle is an optimized lifecycle tool that lets you focus on the most common lifecycle management needs, without unnecessary hardware-centric concepts like data tiers.
+:::
+::::
 
-$$$ilm-gs-alias-bootstrap$$$
+::::{important}
+To use {{ilm-init}}, all nodes in a cluster must run the same version. Although it might be possible to create and apply policies in a mixed-version cluster, there is no guarantee they will work as intended. Attempting to use a policy that contains actions that aren’t supported on all nodes in a cluster will cause errors.
+::::
 
-$$$manage-time-series-data-without-data-streams$$$
+## Actions
 
-$$$ilm-gs-create-policy$$$
+{{ilm-init}} policies can trigger actions like:
 
-$$$ilm-gs-apply-policy$$$
+* **Rollover**: Creates a new write index when the current one reaches a certain size, number of docs, or age.
+* **Shrink**: Reduces the number of primary shards in an index.
+* **Force merge**: Triggers a [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html) to reduce the number of segments in an index’s shards.
+* **Delete**: Permanently remove an index, including all of its data and metadata.
+* [And more](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-actions.html)
 
-$$$ilm-gs-create-the-data-stream$$$
+Each action has options you can use to specify index behavior and characteristics like:
 
-$$$ilm-gs-check-progress$$$
+* The maximum shard size, number of documents, or age at which you want to roll over to a new index.
+* The point at which the index is no longer being updated and the number of primary shards can be reduced.
+* When to force a merge to permanently remove documents marked for deletion.
+* The point at which the index can be moved to less performant hardware.
+* The point at which the availability is not as critical and the number of replicas can be reduced.
+* When the index can be safely deleted.
 
-$$$ilm-gs-alias-apply-policy$$$
+For example, if you are indexing metrics data from a fleet of ATMs into Elasticsearch, you might define a policy that says:
 
-$$$ilm-gs-alias-check-progress$$$
+1. When the total size of the index’s primary shards reaches 50GB, roll over to a new index.
+2. Move the old index into the warm phase, mark it read only, and shrink it down to a single shard.
+3. After 7 days, move the index into the cold phase and move it to less expensive hardware.
+4. Delete the index once the required 30 day retention period is reached.
+
+**Learn about all available actions in [Index lifecycle actions](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-actions.html).**
+
+## Create and manage {{ilm-init}} policies
+
+You can create and manage index lifecycle policies through [{{kib}} Management](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) or the [{{ilm-init}} APIs](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-ilm). For more details on creating and managing index lifecycle policies refer to:
+
+* [Configure a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md)
+* [Update a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md)
+* [Start and stop index lifecycle management](/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md)
+* [Restore a managed data stream or index](/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md)
+* [Customize built-in policies](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md)
+
+Default {{ilm}} policies are created automatically when you use {{agent}}, {{beats}}, or the {{ls}} {{es}} output plugin to send data to the {{stack}}.
+
+![index lifecycle policies](../../images/elasticsearch-reference-index-lifecycle-policies.png)
+
+::::{tip}
+To automatically back up your indices and manage snapshots, use [snapshot lifecycle policies](/deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm).
+::::
+
+## Migrate to {{ilm-init}}
+
+For existing hot-warm deployments that are currently using index curation, migrating to ILM gives you more fine-grained control over the lifecycle of each index. Read more in:
+
+* [Manage existing indices](/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md)
+* [Migrate to index lifecycle management](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md)
+* [Migrate index allocation filters to node roles](/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md)

manage-data/lifecycle/index-lifecycle-management/concepts.md

@@ -1,4 +1,5 @@
 ---
+navigation_title: Configure a policy
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html
 ---
@@ -18,7 +19,6 @@ When you enable {{ilm}} for {{beats}} or the {{ls}} {{es}} output plugin, the ne
 ::::
 
 
-
 ## Create lifecycle policy [ilm-create-policy]
 
 To create a lifecycle policy from {{kib}}, open the menu and go to **Stack Management > Index Lifecycle Policies**. Click **Create policy**.

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

@@ -1,24 +1,28 @@
-# Migrate to index lifecycle management [ec-migrate-index-management]
+---
+mapped_pages:
+  - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrate-index-management.html
+  - https://www.elastic.co/guide/en/cloud/current/ec-configure-index-management.html
+  - https://www.elastic.co/guide/en/cloud/current/ec-migrate-index-management.html
+---
+
+# Migrate to index lifecycle management
 
 ::::{important}
 Index curation is deprecated. Any deployments using index curation are prompted to migrate to ILM.
 ::::
 
-
 The index lifecycle management (ILM) feature of the {{stack}} provides an integrated and streamlined way to manage time-based data, making it easier to follow best practices for managing your indices. Compared to index curation, migrating to ILM gives you more fine-grained control over the lifecycle of each index.
 
 For existing hot-warm deployments that are currently using index curation, there are a couple of options for migrating to index lifecycle management (ILM). You can:
 
 * Use the migration process in the console to change an existing deployment to ILM.
 * Take a snapshot and restore your data to a new Elastic Stack deployment that has ILM enabled.
 
-To learn more about configuring index lifecycle management for Elasticsearch Service or about all of the features that are available with ILM, see:
-
-* [Create your index lifecyle policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html)
-* [Managing the index lifecycle](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html)
-
 To configure ILM Migration in the console:
 
+::::{tab-set}
+
+:::{tab-item} {{ech}}
 1. Log in to the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body).
 2. From the **Deployments** page, select your deployment.
 
@@ -27,12 +31,25 @@ To configure ILM Migration in the console:
 3. Near the top of the deployment overview, you should get a message to migrate from index curation to index lifecycle management (ILM) along with a **Start migration** button.
 4. Select which index curation pattern you wish to migrate.
 5. Set the ILM policy name for each data view (formerly *index pattern*).
+:::
+
+:::{tab-item} Elastic Cloud Enterprise
+1. [Log into the Cloud UI](../../../deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md).
+2. From the **Deployments** page, select your deployment.
+
+    On the deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.
+
+3. Near the top of the deployment overview, you should get a message to migrate from index curation to index lifecycle management (ILM) along with a **Start migration** button.
+4. Select which index curation pattern you wish to migrate.
+5. Set the ILM policy name for each data view (formerly *index pattern*).
+:::
+
+::::
 
 ::::{note}
 Beginning with Elastic Stack version 8.0, Kibana *index patterns* have been renamed to *data views*. To learn more, check the Kibana [What’s new in 8.0](https://www.elastic.co/guide/en/kibana/8.0/whats-new.html#index-pattern-rename) page.
 ::::
 
-
 1. Set the shard allocation attribute for the data view.
 
     * You can set different node attributes per data view to allow for more in-depth configuration in Kibana, or

raw-migrated-files/cloud/cloud/ec-migrate-index-management.md renamed to manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md

@@ -1,12 +1,10 @@
 ---
-navigation_title: "Policy updates"
+navigation_title: "Update a policy"
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/update-lifecycle-policy.html
 ---
 
-
-
-# Policy updates [update-lifecycle-policy]
+# Update a lifecycle policy
 
 
 You can change how the lifecycle of an index or collection of rolling indices is managed by modifying the current policy or switching to a different policy.
@@ -16,7 +14,7 @@ To ensure that policy updates don’t put an index into a state where it can’t
 When the index advances to the next phase, it uses the phase definition from the updated policy.
 
 
-## How changes are applied [ilm-apply-changes] 
+## How changes are applied [ilm-apply-changes]
 
 When a policy is initially applied to an index, the index gets the latest version of the policy. If you update the policy, the policy version is bumped and {{ilm-init}} can detect that the index is using an earlier version that needs to be updated.
 
@@ -25,7 +23,7 @@ Changes to `min_age` are not propagated to the cached definition. Changing a pha
 For example, if you create a policy that has a hot phase that does not specify a `min_age`, indices immediately enter the hot phase when the policy is applied. If you then update the policy to specify a `min_age` of 1 day for the hot phase, that has no effect on indices that are already in the hot phase. Indices created *after* the policy update won’t enter the hot phase until they are a day old.
 
 
-## How new policies are applied [ilm-apply-new-policy] 
+## How new policies are applied [ilm-apply-new-policy]
 
 When you apply a different policy to a managed index, the index completes the current phase using the cached definition from the previous policy. The index starts using the new policy when it moves to the next phase.
 

manage-data/lifecycle/index-lifecycle-management/policy-updates.md

@@ -1,4 +1,5 @@
 ---
+navigation_title: Start and stop {{ilm-init}}
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/start-stop-ilm.html
 ---
@@ -9,13 +10,13 @@ By default, the {{ilm-init}} service is in the `RUNNING` state and manages all i
 
 You can stop {{ilm}} to suspend management operations for all indices. For example, you might stop {{ilm}} when performing scheduled maintenance or making changes to the cluster that could impact the execution of {{ilm-init}} actions.
 
-::::{important} 
+::::{important}
 When you stop {{ilm-init}}, [{{slm-init}}](../../../deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm) operations are also suspended. No snapshots will be taken as scheduled until you restart {{ilm-init}}. In-progress snapshots are not affected.
 ::::
 
 
 
-## Get {{ilm-init}} status [get-ilm-status] 
+## Get {{ilm-init}} status [get-ilm-status]
 
 To see the current status of the {{ilm-init}} service, use the [Get Status API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-get-status):
 
@@ -32,7 +33,7 @@ Under normal operation, the response shows {{ilm-init}} is `RUNNING`:
 ```
 
 
-## Stop {{ilm-init}} [stop-ilm] 
+## Stop {{ilm-init}} [stop-ilm]
 
 To stop the {{ilm-init}} service and pause execution of all lifecycle policies, use the [Stop API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-stop):
 
@@ -57,7 +58,7 @@ Once all policies are at a safe stopping point, {{ilm-init}} moves into the `STO
 ```
 
 
-## Start {{ilm-init}} [_start_ilm_init] 
+## Start {{ilm-init}} [_start_ilm_init]
 
 To restart {{ilm-init}} and resume executing policies, use the [Start API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-start). This puts the  {{ilm-init}} service in the `RUNNING` state and {{ilm-init}} begins executing policies from where it left off.