Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1775,7 +1775,7 @@ Once a deployment is migrated to node roles, it is not possible to roll back.
::::


After the migration plan has finished, we recommend following the [Migrate index allocation filters to node roles](../../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md) guide. Step 1 of this guide was already accomplished by adding support for `node_roles`. However, performing steps 2, 3, and 4, which involves updating index settings, index templates, and ILM policies, can prevent shard allocation issues caused by conflicting ILM policies.
After the migration plan has finished, we recommend following the [Migrate index allocation filters to node roles](../../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md) guide. Step 1 of this guide was already accomplished by adding support for `node_roles`. However, performing steps 2, 3, and 4, which involves updating index settings, index templates, and ILM policies, can prevent shard allocation issues caused by conflicting ILM policies.


### Example [ece_example_2]
Expand Down
4 changes: 2 additions & 2 deletions manage-data/lifecycle/index-lifecycle-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,6 @@ To automatically back up your indices and manage snapshots, use [snapshot lifecy

For existing hot-warm deployments that are currently using index curation, migrating to ILM gives you more fine-grained control over the lifecycle of each index. Read more in:

* [Manage existing indices](/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md)
* [Manage existing indices](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management/manage-existing-indices.md)
* [Migrate to index lifecycle management](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md)
* [Migrate index allocation filters to node roles](/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md)
* [Migrate index allocation filters to node roles](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md)
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Starting in Curator version 5.7, Curator ignores {{ilm-init}} managed indices.

## Apply policies to existing time series indices [ilm-existing-indices-apply]

The simplest way to transition to managing your periodic indices with {{ilm-init}} is to [configure an index template](configure-lifecycle-policy.md#apply-policy-template) to apply a lifecycle policy to new indices. Once the index you are writing to is being managed by {{ilm-init}}, you can [manually apply a policy](configure-lifecycle-policy.md#apply-policy-multiple) to your older indices.
The simplest way to transition to managing your periodic indices with {{ilm-init}} is to [configure an index template](../configure-lifecycle-policy.md#apply-policy-template) to apply a lifecycle policy to new indices. Once the index you are writing to is being managed by {{ilm-init}}, you can [manually apply a policy](../configure-lifecycle-policy.md#apply-policy-multiple) to your older indices.

Define a separate policy for your older indices that omits the rollover action. Rollover is used to manage where new data goes, so isn’t applicable.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ products:

# Migrate index allocation filters to node roles [migrate-index-allocation-filters]

If you currently use [custom node attributes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#custom-node-attributes) and [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) to move indices through [data tiers](../data-tiers.md) in a [hot-warm-cold architecture](https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management), we recommend that you switch to using the built-in node roles and automatic [data tier allocation](../data-tiers.md#data-tier-allocation). Using node roles enables {{ilm-init}} to automatically move indices between data tiers.
If you currently use [custom node attributes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#custom-node-attributes) and [attribute-based allocation filters](../../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) to move indices through [data tiers](../../data-tiers.md) in a [hot-warm-cold architecture](https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management), we recommend that you switch to using the built-in node roles and automatic [data tier allocation](../../data-tiers.md#data-tier-allocation). Using node roles enables {{ilm-init}} to automatically move indices between data tiers.

::::{note}
While we recommend relying on automatic data tier allocation to manage your data in a hot-warm-cold architecture, you can still use attribute-based allocation filters to control shard allocation for other purposes.
Expand All @@ -25,11 +25,11 @@ If you are using node attributes from the default deployment template in {{ech}}

* Upgrade to {{es}} 7.10 or higher
* Deploy a warm, cold, or frozen data tier
* [Enable autoscaling](../../../deploy-manage/autoscaling.md)
* [Enable autoscaling](../../../../deploy-manage/autoscaling.md)

These actions automatically update your cluster configuration and {{ilm-init}} policies to use node roles. Additionally, upgrading to version 7.14 or higher automatically update {{ilm-init}} policies whenever any configuration change is applied to your deployment.

If you use custom index templates, check them after the automatic migration completes and remove any [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md).
If you use custom index templates, check them after the automatic migration completes and remove any [attribute-based allocation filters](../../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md).

::::{note}
You do not need to take any further action after the automatic migration. The following manual steps are only necessary if you do not allow the automatic migration or have a self-managed deployment.
Expand All @@ -51,7 +51,7 @@ To switch to using node roles:

Configure the appropriate roles for each data node to assign it to one or more data tiers: `data_hot`, `data_content`, `data_warm`, `data_cold`, or `data_frozen`. A node can also have other [roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md). By default, new nodes are configured with all roles.

When you add a data tier to an {{ech}} deployment, one or more nodes are automatically configured with the corresponding role. To explicitly change the role of a node in an {{ech}} deployment, use the [Update deployment API](../../../deploy-manage/deploy/elastic-cloud/manage-deployments-using-elastic-cloud-api.md#ec_update_a_deployment). Replace the node’s `node_type` configuration with the appropriate `node_roles`. For example, the following configuration adds the node to the hot and content tiers, and enables it to act as an ingest node, remote, and transform node.
When you add a data tier to an {{ech}} deployment, one or more nodes are automatically configured with the corresponding role. To explicitly change the role of a node in an {{ech}} deployment, use the [Update deployment API](../../../../deploy-manage/deploy/elastic-cloud/manage-deployments-using-elastic-cloud-api.md#ec_update_a_deployment). Replace the node’s `node_type` configuration with the appropriate `node_roles`. For example, the following configuration adds the node to the hot and content tiers, and enables it to act as an ingest node, remote, and transform node.

```yaml
"node_roles": [
Expand Down Expand Up @@ -92,9 +92,9 @@ On {{ech}} deployments, remove the `cloud-hot-warm-allocation-0` index template
DELETE _template/.cloud-hot-warm-allocation-0
```

If you’re using a custom index template, update it to remove the [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) you used to assign new indices to the hot tier.
If you’re using a custom index template, update it to remove the [attribute-based allocation filters](../../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) you used to assign new indices to the hot tier.

To completely avoid the issues that raise when mixing the tier preference and custom attribute routing setting we also recommend updating all the legacy, composable, and component templates to remove the [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) from the settings they configure.
To completely avoid the issues that raise when mixing the tier preference and custom attribute routing setting we also recommend updating all the legacy, composable, and component templates to remove the [attribute-based allocation filters](../../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) from the settings they configure.


### Set a tier preference for existing indices [set-tier-preference]
Expand Down
2 changes: 1 addition & 1 deletion troubleshoot/elasticsearch/add-tier.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ In order to get the shards assigned we need enable a new tier in the deployment.
::::::

::::::{tab-item} Self-managed
In order to get the shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes.
In order to get the shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes.

To determine which tier an index requires for assignment, use the [get index setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) API to retrieve the configured value for the `index.routing.allocation.include._tier_preference` setting:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ In order to get the shards assigned we’ll need to increase the number of shard
::::::

::::::{tab-item} Self-managed
In order to get the shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes.
In order to get the shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes.

To inspect which tier is an index targeting for assignment, use the [get index setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) API to retrieve the configured value for the `index.routing.allocation.include._tier_preference` setting:

Expand Down
2 changes: 1 addition & 1 deletion troubleshoot/elasticsearch/increase-shard-limit.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ In order to get the shards assigned we’ll need to increase the number of shard
::::::

::::::{tab-item} Self-managed
In order to get the shards assigned you can add more nodes to your {{es}} cluster and assing the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes.
In order to get the shards assigned you can add more nodes to your {{es}} cluster and assing the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes.

To inspect which tier is an index targeting for assignment, use the [get index setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) API to retrieve the configured value for the `index.routing.allocation.include._tier_preference` setting:

Expand Down
2 changes: 1 addition & 1 deletion troubleshoot/elasticsearch/increase-tier-capacity.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ If it is not possible to increase the size per zone or the number of availabilit
::::::

::::::{tab-item} Self-managed
In order to get the replica shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes.
In order to get the replica shards assigned you can add more nodes to your {{es}} cluster and assign the index’s target tier [node role](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the new nodes.

To inspect which tier an index is targeting for assignment, use the [get index setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) API to retrieve the configured value for the `index.routing.allocation.include._tier_preference` setting:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ In order to get the shards assigned we need to call the [migrate to data tiers r
::::::{tab-item} Self-managed
In order to get the shards assigned we need to make sure the deployment is using the [data tiers](../../manage-data/lifecycle/data-tiers.md) node roles and then call the [migrate to data tiers routing](../../manage-data/lifecycle/data-tiers.md) API which will resolve the conflicting routing configurations towards using the standardized [data tiers](../../manage-data/lifecycle/data-tiers.md). This will also future-proof the system by migrating the index templates and ILM policies if needed.

1. In case your deployment is not yet using [data tiers](../../manage-data/lifecycle/data-tiers.md) [assign data nodes](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the appropriate data tier. Configure the appropriate roles for each data node to assign it to one or more data tiers: `data_hot`, `data_content`, `data_warm`, `data_cold`, or `data_frozen`. For example, the following setting configures a node to be a data-only node in the hot and content tiers.
1. In case your deployment is not yet using [data tiers](../../manage-data/lifecycle/data-tiers.md) [assign data nodes](../../manage-data/lifecycle/index-lifecycle-management/migrate-index-management/migrate-index-allocation-filters-to-node-roles.md#assign-data-tier) to the appropriate data tier. Configure the appropriate roles for each data node to assign it to one or more data tiers: `data_hot`, `data_content`, `data_warm`, `data_cold`, or `data_frozen`. For example, the following setting configures a node to be a data-only node in the hot and content tiers.

```yaml
node.roles [ data_hot, data_content ]
Expand Down
Loading