diff --git a/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md b/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md index 2566230b98..e7414e53c7 100644 --- a/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md +++ b/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md @@ -1775,7 +1775,7 @@ Once a deployment is migrated to node roles, it is not possible to roll back. :::: -After the migration plan has finished, we recommend following the [Migrate index allocation filters to node roles](../../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md) guide. Step 1 of this guide was already accomplished by adding support for `node_roles`. However, performing steps 2, 3, and 4, which involves updating index settings, index templates, and ILM policies, can prevent shard allocation issues caused by conflicting ILM policies. +After the migration plan has finished, we recommend following the [Migrate index allocation filters to node roles](../../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md) guide. Step 1 of this guide was already accomplished by adding support for `node_roles`. However, performing steps 2, 3, and 4, which involves updating index settings, index templates, and ILM policies, can prevent shard allocation issues caused by conflicting ILM policies. ### Example [ece_example_2] diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index 1f1d81cda2..170f493367 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -83,6 +83,6 @@ To automatically back up your indices and manage snapshots, use [snapshot lifecy For existing hot-warm deployments that are currently using index curation, migrating to ILM gives you more fine-grained control over the lifecycle of each index. Read more in: -* [Manage existing indices](/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md) +* [Manage existing indices](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management/manage-existing-indices.md) * [Migrate to index lifecycle management](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md) -* [Migrate index allocation filters to node roles](/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md) +* [Migrate index allocation filters to node roles](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md) diff --git a/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md b/manage-data/lifecycle/index-lifecycle-management/migrate-index-management/manage-existing-indices.md similarity index 94% rename from manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md rename to manage-data/lifecycle/index-lifecycle-management/migrate-index-management/manage-existing-indices.md index 5c645e8ad4..58929d0393 100644 --- a/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md +++ b/manage-data/lifecycle/index-lifecycle-management/migrate-index-management/manage-existing-indices.md @@ -22,7 +22,7 @@ Starting in Curator version 5.7, Curator ignores {{ilm-init}} managed indices. ## Apply policies to existing time series indices [ilm-existing-indices-apply] -The simplest way to transition to managing your periodic indices with {{ilm-init}} is to [configure an index template](configure-lifecycle-policy.md#apply-policy-template) to apply a lifecycle policy to new indices. Once the index you are writing to is being managed by {{ilm-init}}, you can [manually apply a policy](configure-lifecycle-policy.md#apply-policy-multiple) to your older indices. +The simplest way to transition to managing your periodic indices with {{ilm-init}} is to [configure an index template](../configure-lifecycle-policy.md#apply-policy-template) to apply a lifecycle policy to new indices. Once the index you are writing to is being managed by {{ilm-init}}, you can [manually apply a policy](../configure-lifecycle-policy.md#apply-policy-multiple) to your older indices. Define a separate policy for your older indices that omits the rollover action. Rollover is used to manage where new data goes, so isn’t applicable. diff --git a/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md b/manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md similarity index 83% rename from manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md rename to manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md index 8b7851b3b9..395d8c0271 100644 --- a/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md +++ b/manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md @@ -9,7 +9,7 @@ products: # Migrate index allocation filters to node roles [migrate-index-allocation-filters] -If you currently use [custom node attributes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#custom-node-attributes) and [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) to move indices through [data tiers](../data-tiers.md) in a [hot-warm-cold architecture](https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management), we recommend that you switch to using the built-in node roles and automatic [data tier allocation](../data-tiers.md#data-tier-allocation). Using node roles enables {{ilm-init}} to automatically move indices between data tiers. +If you currently use [custom node attributes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#custom-node-attributes) and [attribute-based allocation filters](../../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) to move indices through [data tiers](../../data-tiers.md) in a [hot-warm-cold architecture](https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management), we recommend that you switch to using the built-in node roles and automatic [data tier allocation](../../data-tiers.md#data-tier-allocation). Using node roles enables {{ilm-init}} to automatically move indices between data tiers. ::::{note} While we recommend relying on automatic data tier allocation to manage your data in a hot-warm-cold architecture, you can still use attribute-based allocation filters to control shard allocation for other purposes. @@ -25,11 +25,11 @@ If you are using node attributes from the default deployment template in {{ech}} * Upgrade to {{es}} 7.10 or higher * Deploy a warm, cold, or frozen data tier -* [Enable autoscaling](../../../deploy-manage/autoscaling.md) +* [Enable autoscaling](../../../../deploy-manage/autoscaling.md) These actions automatically update your cluster configuration and {{ilm-init}} policies to use node roles. Additionally, upgrading to version 7.14 or higher automatically update {{ilm-init}} policies whenever any configuration change is applied to your deployment. -If you use custom index templates, check them after the automatic migration completes and remove any [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md). +If you use custom index templates, check them after the automatic migration completes and remove any [attribute-based allocation filters](../../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md). ::::{note} You do not need to take any further action after the automatic migration. The following manual steps are only necessary if you do not allow the automatic migration or have a self-managed deployment. @@ -51,7 +51,7 @@ To switch to using node roles: Configure the appropriate roles for each data node to assign it to one or more data tiers: `data_hot`, `data_content`, `data_warm`, `data_cold`, or `data_frozen`. A node can also have other [roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md). By default, new nodes are configured with all roles. -When you add a data tier to an {{ech}} deployment, one or more nodes are automatically configured with the corresponding role. To explicitly change the role of a node in an {{ech}} deployment, use the [Update deployment API](../../../deploy-manage/deploy/elastic-cloud/manage-deployments-using-elastic-cloud-api.md#ec_update_a_deployment). Replace the node’s `node_type` configuration with the appropriate `node_roles`. For example, the following configuration adds the node to the hot and content tiers, and enables it to act as an ingest node, remote, and transform node. +When you add a data tier to an {{ech}} deployment, one or more nodes are automatically configured with the corresponding role. To explicitly change the role of a node in an {{ech}} deployment, use the [Update deployment API](../../../../deploy-manage/deploy/elastic-cloud/manage-deployments-using-elastic-cloud-api.md#ec_update_a_deployment). Replace the node’s `node_type` configuration with the appropriate `node_roles`. For example, the following configuration adds the node to the hot and content tiers, and enables it to act as an ingest node, remote, and transform node. ```yaml "node_roles": [ @@ -92,9 +92,9 @@ On {{ech}} deployments, remove the `cloud-hot-warm-allocation-0` index template DELETE _template/.cloud-hot-warm-allocation-0 ``` -If you’re using a custom index template, update it to remove the [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) you used to assign new indices to the hot tier. +If you’re using a custom index template, update it to remove the [attribute-based allocation filters](../../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) you used to assign new indices to the hot tier. -To completely avoid the issues that raise when mixing the tier preference and custom attribute routing setting we also recommend updating all the legacy, composable, and component templates to remove the [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) from the settings they configure. +To completely avoid the issues that raise when mixing the tier preference and custom attribute routing setting we also recommend updating all the legacy, composable, and component templates to remove the [attribute-based allocation filters](../../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) from the settings they configure. ### Set a tier preference for existing indices [set-tier-preference] diff --git a/troubleshoot/elasticsearch/add-tier.md b/troubleshoot/elasticsearch/add-tier.md index 2b2fc7ae1b..3068d6334c 100644 --- a/troubleshoot/elasticsearch/add-tier.md +++ b/troubleshoot/elasticsearch/add-tier.md @@ -71,7 +71,7 @@ In order to get the shards assigned we need enable a new tier in the deployment. :::::: ::::::{tab-item} Self-managed -In order to get the shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes. +In order to get the shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes. To determine which tier an index requires for assignment, use the [get index setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) API to retrieve the configured value for the `index.routing.allocation.include._tier_preference` setting: diff --git a/troubleshoot/elasticsearch/increase-cluster-shard-limit.md b/troubleshoot/elasticsearch/increase-cluster-shard-limit.md index c9d836460a..60d5d17b34 100644 --- a/troubleshoot/elasticsearch/increase-cluster-shard-limit.md +++ b/troubleshoot/elasticsearch/increase-cluster-shard-limit.md @@ -78,7 +78,7 @@ In order to get the shards assigned we’ll need to increase the number of shard :::::: ::::::{tab-item} Self-managed -In order to get the shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes. +In order to get the shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes. To inspect which tier is an index targeting for assignment, use the [get index setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) API to retrieve the configured value for the `index.routing.allocation.include._tier_preference` setting: diff --git a/troubleshoot/elasticsearch/increase-shard-limit.md b/troubleshoot/elasticsearch/increase-shard-limit.md index 9a5357cf96..572f299d53 100644 --- a/troubleshoot/elasticsearch/increase-shard-limit.md +++ b/troubleshoot/elasticsearch/increase-shard-limit.md @@ -79,7 +79,7 @@ In order to get the shards assigned we’ll need to increase the number of shard :::::: ::::::{tab-item} Self-managed -In order to get the shards assigned you can add more nodes to your {{es}} cluster and assing the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes. +In order to get the shards assigned you can add more nodes to your {{es}} cluster and assing the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes. To inspect which tier is an index targeting for assignment, use the [get index setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) API to retrieve the configured value for the `index.routing.allocation.include._tier_preference` setting: diff --git a/troubleshoot/elasticsearch/increase-tier-capacity.md b/troubleshoot/elasticsearch/increase-tier-capacity.md index 711989b15f..4b1dc7df53 100644 --- a/troubleshoot/elasticsearch/increase-tier-capacity.md +++ b/troubleshoot/elasticsearch/increase-tier-capacity.md @@ -136,7 +136,7 @@ If it is not possible to increase the size per zone or the number of availabilit :::::: ::::::{tab-item} Self-managed -In order to get the replica shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes. +In order to get the replica shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes. To inspect which tier an index is targeting for assignment, use the [get index setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) API to retrieve the configured value for the `index.routing.allocation.include._tier_preference` setting: diff --git a/troubleshoot/elasticsearch/troubleshoot-migrate-to-tiers.md b/troubleshoot/elasticsearch/troubleshoot-migrate-to-tiers.md index fd4b687902..1da5c2e8ac 100644 --- a/troubleshoot/elasticsearch/troubleshoot-migrate-to-tiers.md +++ b/troubleshoot/elasticsearch/troubleshoot-migrate-to-tiers.md @@ -115,7 +115,7 @@ In order to get the shards assigned we need to call the [migrate to data tiers r ::::::{tab-item} Self-managed In order to get the shards assigned we need to make sure the deployment is using the [data tiers](../../manage-data/lifecycle/data-tiers.md) node roles and then call the [migrate to data tiers routing](../../manage-data/lifecycle/data-tiers.md) API which will resolve the conflicting routing configurations towards using the standardized [data tiers](../../manage-data/lifecycle/data-tiers.md). This will also future-proof the system by migrating the index templates and ILM policies if needed. -1. In case your deployment is not yet using [data tiers](../../manage-data/lifecycle/data-tiers.md) [assign data nodes](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the appropriate data tier. Configure the appropriate roles for each data node to assign it to one or more data tiers: `data_hot`, `data_content`, `data_warm`, `data_cold`, or `data_frozen`. For example, the following setting configures a node to be a data-only node in the hot and content tiers. +1. In case your deployment is not yet using [data tiers](../../manage-data/lifecycle/data-tiers.md) [assign data nodes](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the appropriate data tier. Configure the appropriate roles for each data node to assign it to one or more data tiers: `data_hot`, `data_content`, `data_warm`, `data_cold`, or `data_frozen`. For example, the following setting configures a node to be a data-only node in the hot and content tiers. ```yaml node.roles [ data_hot, data_content ]