diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md b/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md index 1df60d8c18..f324792c7c 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md @@ -13,54 +13,60 @@ products: If you create a deployment template that includes more than one data configuration, you must also specify how {{ece}} should manage indices for your users when they create their deployments. For time-series use cases such as logging, metrics, and APM, providing a template that enables index management ensures that data is being stored in the most cost-effective way possible as it ages. -In a template that creates a hot-warm architecture, you can use index curation to specify where new indices are created initially and where they are moved to later on. However, index curation has been deprecated in favor of index lifecycle management, which offers additional features and more fine-grained control over indices. For instance, using ILM you can enable automatic roll-over of index aliases to new indices when existing indices become too large or too old, and you can set indices to be deleted when they are no longer useful. +Configuring index management is part of the larger task of [creating deployment templates](ece-configuring-ece-create-templates.md) or editing them. The choices you make here determine which index management methods are available to your users when they create deployments. +You should configure all index management methods that you want your users to be able to choose from when they create their deployments from your template. You can configure index curation, index lifecycle management, or both. -## Before you begin [ece_before_you_begin_4] -Configuring index management is part of the larger task of [creating deployment templates](ece-configuring-ece-create-templates.md) or editing them. The choices you make here determine which index management methods are available to your users when they create deployments. +## Available index management strategies -You should configure all index management methods that you want your users to be able to choose from when they create their deployments from your template. You can configure index curation, index lifecycle management, or both. +Index lifecycle management +: Uses the [{{ilm}}](/manage-data/lifecycle/index-lifecycle-management.md) feature of the {{stack}} that provides an integrated and streamlined way to manage time-based data, making it easier to follow best practices for managing your indices. Compared to index curation, ILM gives you more fine-grained control over the lifecycle of each index. -::::{note} -Index curation has been deprecated in favor of [index lifecycle management](/manage-data/lifecycle/index-lifecycle-management.md). For {{stack}} version 6.7 and later, any deployments using index curation will be prompted to [migrate to ILM](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md). -:::: +Index curation (Curator) {applies_to}`stack: deprecated 6.7` +: Creates new indices on hot nodes first and moves them to warm nodes later on, based on the data views (formerly *index patterns*) you specify. Also manages replica counts for you, so that all shards of an index can fit on the right data nodes. Compared to index lifecycle management, index curation for time-based indices supports only one action, to move indices from nodes on one data configuration to another, but it is more straightforward to set up initially and all setup can be done directly from the Cloud UI. + + If your users need to delete indices once they are no longer useful to them, they can run [Curator](curator://reference/index.md) on-premise to manage indices for {{es}} clusters hosted on {{ece}}. -## Steps [ece_steps_2] + ::::{note} + Index curation has been deprecated in favor of [index lifecycle management](/manage-data/lifecycle/index-lifecycle-management.md). For {{stack}} version 6.7 and later, any deployments using index curation will be prompted to [migrate to ILM](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md). + :::: -To configure index management when you create a deployment template: -1. On the **Index Management** page, configure the index curation methods that you want to be available when your users create deployments: +## Configure {{ilm}} - Index lifecycle management - : Uses the ILM feature of the {{stack}} that provides an integrated and streamlined way to manage time-based data, making it easier to follow best practices for managing your indices. Compared to index curation, ILM gives you more fine-grained control over the lifecycle of each index. +To configure {{ilm}} as part of your deployment template: - To configure index lifecycle management: +On the **Index Management** page, under **{{ilm}} ({{ilm-init}})**, specify the node attributes for your data configurations. - 1. Specify the node attributes for your data configurations. +Node attributes are simple key-value pairs, such as `node_type: hot`, `node_type: warm`, and `node_type: cold`. These node attributes add defining metadata attributes to each data configuration in your template that tell your users what they can be used for. What you define here should help guide your users when they set up their index lifecycle management policy in {{kib}}, such as a hot-warm policy. - Node attributes are simple key-value pairs, such as `node_type: hot`, `node_type: warm`, and `node_type: cold`. These node attributes add defining metadata attributes to each data configuration in your template that tell your users what they can be used for. What you define here should help guide your users when they set up their index lifecycle management policy in {{kib}}, such as a hot-warm policy. +For each data tier, specify an attribute key-value pair in the **Node attributes** field, with the key and value separated by a colon. Repeat this process until you have added all the node attributes that you want to be available to your users when they create an index lifecycle policy later on. - 1. Specify an attribute key-value pair in the **Node attributes** field, with the key and value separated by a colon. - 2. Repeat the previous step until you have added all the node attributes that you want to be available to your users when they create an index lifecycle policy later on. +## Configure index curation +```{applies_to} +stack: deprecated 6.7 +``` +::::{note} +Index curation has been deprecated in favor of [index lifecycle management](/manage-data/lifecycle/index-lifecycle-management.md). For {{stack}} version 6.7 and later, any deployments using index curation will be prompted to [migrate to ILM](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md). +:::: - Index curation - : Creates new indices on hot nodes first and moves them to warm nodes later on, based on the data views (formerly *index patterns*) you specify. Also manages replica counts for you, so that all shards of an index can fit on the right data nodes. Compared to index lifecycle management, index curation for time-based indices supports only one action, to move indices from nodes on one data configuration to another, but it is more straightforward to set up initially and all setup can be done directly from the Cloud UI. +To configure index curation as part of your deployment template: - If your user need to delete indices once they are no longer useful to them, they can run [Curator](curator://reference/index.md) on-premise to manage indices for {{es}} clusters hosted on {{ece}}. +1. On the **Index Management** page, under **Index curation**, click **Configure**. - To configure index curation: +2. Configure index curation by adding an index pattern: - 1. Select the hot data configuration where new indices get created initially. - 2. Select the warm nodes where older indices get moved to later on when they get curated. - 3. Specify which indices get curated by including at least one data view. + 1. Select the hot data configuration where new indices get created initially. + 2. Select the warm nodes where older indices get moved to later on when they get curated. + 3. Specify which indices get curated by including at least one index pattern. - By default, the pattern is `*`, which means that all indices get curated. For logging use cases, you could specify to curate only the `logstash-*`, `metricbeat-*`, or `filebeat-*` data views, for example. + By default, the pattern is `*`, which means that all indices get curated. For logging use cases, you could specify to curate only the `logstash-*`, `metricbeat-*`, or `filebeat-*` data views, for example. - 4. Specify the time interval after which indices get curated. + 4. Specify the time interval after which indices get curated. -2. Select **Next**. +## Next steps After you have completed these steps, continue with [creating your deployment template](ece-configuring-ece-create-templates.md#ece-configuring-ece-create-templates-ui). diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md index 869bbae0e4..b7cbb196d8 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md @@ -75,13 +75,17 @@ Before you start creating your own deployment templates, you should have: [tagge If a data tier or component is not required for your particular use case, you can simply set its initial size per zone to `0`. You can enable a tier or component anytime you need it just by scaling up the size. If autoscaling is enabled, data tiers and machine learning nodes are sized up automatically when they’re needed. For example, when you configure your first machine learning job, ML nodes are enabled by the autoscaling process. Similarly, if you choose to create a cold data phase as part of your deployment’s index lifecycle management (ILM) policy, a cold data node is enabled automatically without your needing to configure it. 8. Select **Manage indices**. -9. On this page you can [configure index management](ece-configure-templates-index-management.md) by assigning attributes to each of the data nodes in the deployment template. In {{kib}}, you can configure an index lifecycle management (ILM) policy, based on the node attributes, to control how data moves across the nodes in your deployment. -10. Select **Stack features**. -11. You can select a [snapshot repository](../../tools/snapshot-and-restore/cloud-enterprise.md) to be used by default for deployment backups. -12. You can choose to [enable logging and monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) by default, so that deployment logs and metrics are send to a dedicated monitoring deployment, and so that additional log types, retention options, and {{kib}} visualizations are available on all deployments created using this template. -13. Select **Extensions**. -14. Select any {{es}} extensions that you would like to be available automatically to all deployments created using the template. -15. Select **Save and create template**. + + On this page, you can [configure index management](ece-configure-templates-index-management.md) by assigning attributes to each of the data nodes in the deployment template. In {{kib}}, you can configure an index lifecycle management (ILM) policy, based on the node attributes, to control how data moves across the nodes in your deployment. +10. Select **Stack features**. On this page, you can manage the following options: + + * You can select a [snapshot repository](../../tools/snapshot-and-restore/cloud-enterprise.md) to be used by default for deployment backups. + + * You can choose to [enable logging and monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) by default, so that deployment logs and metrics are send to a dedicated monitoring deployment, and so that additional log types, retention options, and {{kib}} visualizations are available on all deployments created using this template. +11. Select **Extensions**. + + Select any {{es}} extensions that you would like to be available automatically to all deployments created using the template. +12. Select **Save and create template**. ## Create deployment templates through the RESTful API [ece_create_deployment_templates_through_the_restful_api] @@ -408,14 +412,10 @@ Before you start creating your own deployment templates, you should have: [tagge When specifying `node_roles` in the {{es}} plan of the deployment template, the template must contain all resource types and all {{es}} tiers. The deployment template must contain exactly one entry for each resource type. It must have one {{es}}, one {{kib}}, and one APM. On top of that, it must also include all supported {{es}} tiers in the {{es}} plan. The supported tiers are identified by the IDs `hot_content`, `warm`, `cold`, `frozen`, `master`, `coordinating` and `ml`. :::: - ::::{note} Deployment templates without `node_roles` or `id` should only contain hot and warm data tiers, with different `instance_configuration_id`s. Node roles are highly recommended when using the cold tier and are mandatory for the frozen tier. :::: - -After you have saved your new template, you can start [creating new deployments](create-deployment.md) with it. - ::::{note} To support deployment templates that are versioned due to a constraint on architecture that is only supported by newer versions of ECE, for example ARM instances, you must add additional configuration: @@ -427,3 +427,8 @@ These attributes are set at the same level as `name` and `description`. The UI s :::: +## Next steps + +After you have saved your new template, you can start [creating new deployments](create-deployment.md) with it. + +