Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion deploy-manage/autoscaling/ec-autoscaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ When past behavior on a hot tier indicates that the influx of data can increase

* Through ILM policies. For example, if a deployment has only hot nodes and autoscaling is enabled, it automatically creates warm or cold nodes, if an ILM policy is trying to move data from hot to warm or cold nodes.

On machine learning nodes, scaling is determined by an estimate of the memory and CPU requirements for the currently configured jobs and trained models. When a new machine learning job tries to start, it looks for a node with adequate native memory and CPU capacity. If one cannot be found, it stays in an `opening` state. If this waiting job exceeds the queueing limit set in the machine learning decider, a scale up is requested. Conversely, as machine learning jobs run, their memory and CPU usage might decrease or other running jobs might finish or close. In this case, if the duration of decreased resource usage exceeds the set value for `down_scale_delay`, a scale down is requested. Check [Machine learning decider](autoscaling-deciders.md) for more detail. To learn more about machine learning jobs in general, check [Create anomaly detection jobs](https://www.elastic.co/guide/en/machine-learning/current/create-jobs.html).
On machine learning nodes, scaling is determined by an estimate of the memory and CPU requirements for the currently configured jobs and trained models. When a new machine learning job tries to start, it looks for a node with adequate native memory and CPU capacity. If one cannot be found, it stays in an `opening` state. If this waiting job exceeds the queueing limit set in the machine learning decider, a scale up is requested. Conversely, as machine learning jobs run, their memory and CPU usage might decrease or other running jobs might finish or close. In this case, if the duration of decreased resource usage exceeds the set value for `down_scale_delay`, a scale down is requested. Check [Machine learning decider](autoscaling-deciders.md) for more detail. To learn more about machine learning jobs in general, check [Create anomaly detection jobs](https://www.elastic.co/guide/en/machine-learning/current/ml-ad-run-jobs.html).

On a highly available deployment, autoscaling events are always applied to instances in each availability zone simultaneously, to ensure consistency.

Expand Down
4 changes: 2 additions & 2 deletions deploy-manage/autoscaling/ece-autoscaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ When past behavior on a hot tier indicates that the influx of data can increase

* Through ILM policies. For example, if a deployment has only hot nodes and autoscaling is enabled, it automatically creates warm or cold nodes, if an ILM policy is trying to move data from hot to warm or cold nodes.

On machine learning nodes, scaling is determined by an estimate of the memory and CPU requirements for the currently configured jobs and trained models. When a new machine learning job tries to start, it looks for a node with adequate native memory and CPU capacity. If one cannot be found, it stays in an `opening` state. If this waiting job exceeds the queueing limit set in the machine learning decider, a scale up is requested. Conversely, as machine learning jobs run, their memory and CPU usage might decrease or other running jobs might finish or close. In this case, if the duration of decreased resource usage exceeds the set value for `down_scale_delay`, a scale down is requested. Check [Machine learning decider](autoscaling-deciders.md) for more detail. To learn more about machine learning jobs in general, check [Create anomaly detection jobs](https://www.elastic.co/guide/en/machine-learning/current/create-jobs.html).
On machine learning nodes, scaling is determined by an estimate of the memory and CPU requirements for the currently configured jobs and trained models. When a new machine learning job tries to start, it looks for a node with adequate native memory and CPU capacity. If one cannot be found, it stays in an `opening` state. If this waiting job exceeds the queueing limit set in the machine learning decider, a scale up is requested. Conversely, as machine learning jobs run, their memory and CPU usage might decrease or other running jobs might finish or close. In this case, if the duration of decreased resource usage exceeds the set value for `down_scale_delay`, a scale down is requested. Check [Machine learning decider](autoscaling-deciders.md) for more detail. To learn more about machine learning jobs in general, check [Create anomaly detection jobs](https://www.elastic.co/guide/en/machine-learning/current/ml-ad-run-jobs.html).

On a highly available deployment, autoscaling events are always applied to instances in each availability zone simultaneously, to ensure consistency.

Expand All @@ -79,7 +79,7 @@ A warning is also issued in the ECE `service-constructor` logs with the field `l
The following are known limitations and restrictions with autoscaling:

* Autoscaling will not run if the cluster is unhealthy or if the last Elasticsearch plan failed.
* In the event that an override is set for the instance size or disk quota multiplier for an instance by means of the [Instance Overrides API](https://www.elastic.co/guide/en/cloud-enterprise/current/set-all-instances-settings-overrides.html), autoscaling will be effectively disabled. It’s recommended to avoid adjusting the instance size or disk quota multiplier for an instance that uses autoscaling, since the setting prevents autoscaling.
* In the event that an override is set for the instance size or disk quota multiplier for an instance by means of the [Instance Overrides API](https://www.elastic.co/docs/api/doc/cloud-enterprise/operation/operation-set-all-instances-settings-overrides), autoscaling will be effectively disabled. It’s recommended to avoid adjusting the instance size or disk quota multiplier for an instance that uses autoscaling, since the setting prevents autoscaling.


## Enable or disable autoscaling [ece-autoscaling-enable]
Expand Down
2 changes: 1 addition & 1 deletion deploy-manage/autoscaling/ech-autoscaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ When past behavior on a hot tier indicates that the influx of data can increase

* Through ILM policies. For example, if a deployment has only hot nodes and autoscaling is enabled, it automatically creates warm or cold nodes, if an ILM policy is trying to move data from hot to warm or cold nodes.

On machine learning nodes, scaling is determined by an estimate of the memory and CPU requirements for the currently configured jobs and trained models. When a new machine learning job tries to start, it looks for a node with adequate native memory and CPU capacity. If one cannot be found, it stays in an `opening` state. If this waiting job exceeds the queueing limit set in the machine learning decider, a scale up is requested. Conversely, as machine learning jobs run, their memory and CPU usage might decrease or other running jobs might finish or close. In this case, if the duration of decreased resource usage exceeds the set value for `down_scale_delay`, a scale down is requested. Check [Machine learning decider](autoscaling-deciders.md) for more detail. To learn more about machine learning jobs in general, check [Create anomaly detection jobs](https://www.elastic.co/guide/en/machine-learning/current/create-jobs.html).
On machine learning nodes, scaling is determined by an estimate of the memory and CPU requirements for the currently configured jobs and trained models. When a new machine learning job tries to start, it looks for a node with adequate native memory and CPU capacity. If one cannot be found, it stays in an `opening` state. If this waiting job exceeds the queueing limit set in the machine learning decider, a scale up is requested. Conversely, as machine learning jobs run, their memory and CPU usage might decrease or other running jobs might finish or close. In this case, if the duration of decreased resource usage exceeds the set value for `down_scale_delay`, a scale down is requested. Check [Machine learning decider](autoscaling-deciders.md) for more detail. To learn more about machine learning jobs in general, check [Create anomaly detection jobs](https://www.elastic.co/guide/en/machine-learning/current/ml-ad-run-jobs.html).

On a highly available deployment, autoscaling events are always applied to instances in each availability zone simultaneously, to ensure consistency.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Data inter-node charges are currently waived for Azure deployments.

Data transfer out of deployments and between nodes of the cluster is hard to control, as it is a function of the use case employed for the cluster and cannot always be tuned. Use cases such as batch queries executed at a frequent interval may be revisited to help lower transfer costs, if applicable. Watcher email alerts also count towards data transfer out of the deployment, so you may want to reduce their frequency and size.

The largest contributor to inter-node data transfer is usually shard movement between nodes in a cluster. The only way to prevent shard movement is by having a single node in a single availability zone. This solution is only possible for clusters up to 64GB RAM and is not recommended as it creates a risk of data loss. [Oversharding](https://www.elastic.co/guide/en/elasticsearch/reference/current/avoid-oversharding.html) can cause excessive shard movement. Avoiding oversharding can also help control costs and improve performance. Note that creating snapshots generates inter-node data transfer. The *storage* cost of snapshots is detailed later in this document.
The largest contributor to inter-node data transfer is usually shard movement between nodes in a cluster. The only way to prevent shard movement is by having a single node in a single availability zone. This solution is only possible for clusters up to 64GB RAM and is not recommended as it creates a risk of data loss. [Oversharding](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html) can cause excessive shard movement. Avoiding oversharding can also help control costs and improve performance. Note that creating snapshots generates inter-node data transfer. The *storage* cost of snapshots is detailed later in this document.

The exact root cause of unusual data transfer is not always something we can identify as it can have many causes, some of which are out of our control and not associated with Cloud configuration changes. It may help to [enable monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) and examine index and shard activity on your cluster.

Expand Down
2 changes: 1 addition & 1 deletion deploy-manage/cloud-organization/tools-and-apis.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Most Elastic resources can be accessed and managed through RESTful APIs. While t
Elasticsearch Service API
: You can use the Elasticsearch Service API to manage your deployments and all of the resources associated with them. This includes performing deployment CRUD operations, scaling or autoscaling resources, and managing traffic filters, deployment extensions, remote clusters, and Elastic Stack versions. You can also access cost data by deployment and by organization.

To learn more about the Elasticsearch Service API, read through the [API overview](https://www.elastic.co/guide/en/cloud/current/ec-restful-api.html), try out some [getting started examples](https://www.elastic.co/guide/en/cloud/current/ec-api-examples.html), and check our [API reference documentation](https://www.elastic.co/guide/en/cloud/current/ec-api-swagger.html).
To learn more about the Elasticsearch Service API, read through the [API overview](https://www.elastic.co/guide/en/cloud/current/ec-restful-api.html), try out some [getting started examples](https://www.elastic.co/guide/en/cloud/current/ec-api-examples.html), and check our [API reference documentation](https://www.elastic.co/docs/api/doc/cloud).

Calls to the Elasticsearch Service API are subject to [Rate limiting](https://www.elastic.co/guide/en/cloud/current/ec-api-rate-limiting.html).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1414,7 +1414,7 @@ Having added support for `node_roles` and autoscaling to your custom template, i
curl -k -X GET -H "Authorization: ApiKey $ECE_API_KEY" https://COORDINATOR_HOST:12443/api/v1/deployments/templates?region=ece-region
```

2. Send a `PUT` request with the updated template on the payload, in order to effectively replace the outdated template with the new one. Note that the following request is just an example, you have to replace `{{template_id}}` with the `id` you collected on step 1. and set the payload to the updated template JSON. Check [set deployment template API](https://www.elastic.co/guide/en/cloud-enterprise/current/set-deployment-template-v2.html) for more details.
2. Send a `PUT` request with the updated template on the payload, in order to effectively replace the outdated template with the new one. Note that the following request is just an example, you have to replace `{{template_id}}` with the `id` you collected on step 1. and set the payload to the updated template JSON. Check [set deployment template API](https://www.elastic.co/docs/api/doc/cloud-enterprise/operation/operation-set-deployment-template-v2) for more details.

::::{dropdown} Update template API request example
```sh
Expand Down Expand Up @@ -1749,7 +1749,7 @@ If you do not intend to perform any of these actions, the migration can only be
1. Go to the deployment **Edit** page.
2. Get the deployment update payload by clicking **Equivalent API request** at the bottom of the page.
3. Update the payload by replacing `node_type` with `node_roles` in each Elasticsearch topology element. To know which `node_roles` to add to each topology element, refer to the [custom template example](#ece-ce-add-support-to-node-roles-example) where support for `node_roles` is added.
4. Send a `PUT` request with the updated deployment payload to conclude the migration. Check the [Update Deployment](https://www.elastic.co/guide/en/cloud-enterprise/current/update-deployment.html) API documentation for more details.
4. Send a `PUT` request with the updated deployment payload to conclude the migration. Check the [Update Deployment](https://www.elastic.co/docs/api/doc/cloud-enterprise/operation/operation-update-deployment) API documentation for more details.

**Using the Advanced edit:**

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Create a RHEL 8 (the version must be >= 8.5, but <9), RHEL 9, Rocky Linux 8, or

* For RHEL 8, follow your internal guidelines to add a vanilla RHEL 8 VM to your environment. Note that the version must be >= 8.5, but <9.

Verify that required traffic is allowed. Check the [Networking prerequisites](ece-networking-prereq.md) and [Google Cloud Platform (GCP)](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-gcp.html) guidelines for a list of ports that need to be open. The technical configuration highly depends on the underlying infrastructure.
Verify that required traffic is allowed. Check the [Networking prerequisites](ece-networking-prereq.md) and [Google Cloud Platform (GCP)](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-prereqs.html) guidelines for a list of ports that need to be open. The technical configuration highly depends on the underlying infrastructure.

**Example:** For AWS, allowing traffic between hosts is implemented using security groups.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Send a `PUT` request with the updated template in the payload to replace the ori
* The following request is just an example; other resources in the request payload should remain unchanged (they have been truncated in the example).
* You need to replace `{{template_id}}` in the URL with the `id` that you collected in Step 1.

Refer to [set deployment template API](https://www.elastic.co/guide/en/cloud-enterprise/current/set-deployment-template-v2.html) for more details.
Refer to [set deployment template API](https://www.elastic.co/docs/api/doc/cloud-enterprise/operation/operation-set-deployment-template-v2) for more details.

::::{dropdown} Update template API request example
```sh
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ The API user must have the `Platform admin` role in order to configure system te
```

2. Edit the JSON of the system deployment template you wish to modify.
3. Make the API call to modify the deployment template. Note that the last path segment in the URL is the `id` of the system template you wish to modify. Check [set deployment template API](https://www.elastic.co/guide/en/cloud-enterprise/current/set-deployment-template-v2.html) for more detail.
3. Make the API call to modify the deployment template. Note that the last path segment in the URL is the `id` of the system template you wish to modify. Check [set deployment template API](https://www.elastic.co/docs/api/doc/cloud-enterprise/operation/operation-set-deployment-template-v2) for more detail.

The following example modifies the Default system deployment template (that, is the system template with `id` value of `default`), setting the default value of `autoscaling_enabled` to `true` and the default autoscaling maximum size of the hot tier to 4,194,304MB (64GB * 64 nodes).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,5 +99,5 @@ While the `TransportClient` is deprecated, your custom endpoint aliases still wo
```


For more information on configuring the `TransportClient`, see [Configure the Java Transport Client](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-security-transport.html).
For more information on configuring the `TransportClient`, see [Configure the Java Transport Client](https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/index.html).

Loading
Loading