You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The autoscaling feature adjusts resources based on demand. A deployment can use autoscaling to scale resources as needed, ensuring sufficient capacity to meet workload requirements. In {{ece}}, {{eck}}, and {{ech}} deployments, autoscaling follows predefined policies, while in {{serverless-short}}, it is fully managed and automatic.
Cluster autoscaling allows an operator to create tiers of nodes that monitor themselves and determine if scaling is needed based on an operator-defined policy. An Elasticsearch cluster can use the autoscaling API to report when additional resources are required. For example, an operator can define a policy that scales a warm tier based on available disk space. Elasticsearch monitors disk space in the warm tier. If it predicts low disk space for current and future shard copies, the autoscaling API reports that the cluster needs to scale. It remains the responsibility of the operator to add the additional resources that the cluster signals it requires.
26
27
27
-
% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):
28
+
A policy is composed of a list of roles and a list of deciders. The policy governs the nodes matching the roles. The deciders provide independent estimates of the capacity required. See [Autoscaling deciders](../deploy-manage/autoscaling/autoscaling-deciders.md) for details on available deciders.
28
29
29
-
$$$ec-autoscaling-intro$$$
30
+
Cluster autoscaling supports:
31
+
* Scaling machine learning nodes up and down.
32
+
* Scaling data nodes up based on storage.
30
33
31
-
$$$ec-autoscaling-factors$$$
34
+
## Trained model autoscaling
32
35
33
-
$$$ec-autoscaling-notifications$$$
36
+
:::{admonition} Trained model auto-scaling for self-managed deployments
37
+
The available resources of self-managed deployments are static, so trained model autoscaling is not applicable. However, available resources are still segmented based on the settings described in this section.
38
+
:::
34
39
35
-
$$$ec-autoscaling-restrictions$$$
40
+
Trained model autoscaling automatically adjusts the resources allocated to trained model deployments based on demand. This feature is available on all cloud deployments (ECE, ECK, ECH) and {{serverless-short}}. See [Trained model autoscaling](/deploy-manage/autoscaling/trained-model-autoscaling.md) for details.
36
41
37
-
$$$ec-autoscaling-enable$$$
42
+
Trained model autoscaling supports:
43
+
* Scaling trained model deployments
38
44
39
-
$$$ec-autoscaling-update$$$
45
+
::::{note}
46
+
Autoscaling is not supported on Debian 8.
47
+
::::
40
48
41
-
$$$ece-autoscaling-intro$$$
49
+
Find instructions on setting up and managing autoscaling, including supported environments, configuration options, and examples:
42
50
43
-
$$$ece-autoscaling-factors$$$
44
-
45
-
$$$ece-autoscaling-notifications$$$
46
-
47
-
$$$ece-autoscaling-restrictions$$$
48
-
49
-
$$$ece-autoscaling-enable$$$
50
-
51
-
$$$ece-autoscaling-update$$$
52
-
53
-
$$$ech-autoscaling-intro$$$
54
-
55
-
$$$ech-autoscaling-factors$$$
56
-
57
-
$$$ech-autoscaling-notifications$$$
58
-
59
-
$$$ech-autoscaling-restrictions$$$
60
-
61
-
$$$ech-autoscaling-enable$$$
62
-
63
-
$$$ech-autoscaling-update$$$
64
-
65
-
**This page is a work in progress.** The documentation team is working to combine content pulled from the following pages:
[Autoscaling](/deploy-manage/autoscaling.md) in Elasticsearch enables dynamic resource allocation based on predefined policies. A key component of this mechanism is autoscaling deciders, which independently assess resource requirements and determine when scaling actions are necessary. Deciders analyze various factors, such as storage usage, indexing rates, and machine learning workloads, to ensure clusters maintain optimal performance without manual intervention.
: Estimates required storage capacity as a percentage of the total data set of partially mounted indices. Available for policies governing frozen data nodes.
33
36
34
-
The documentation team is working to combine content pulled from the following pages:
37
+
[Frozen existence decider](#autoscaling-frozen-existence-decider)
38
+
: Estimates a minimum require frozen memory and storage capacity when any index is in the frozen [ILM](../../manage-data/lifecycle/index-lifecycle-management.md) phase.
The [autoscaling](../../deploy-manage/autoscaling.md) reactive storage decider (`reactive_storage`) calculates the storage required to contain the current data set. It signals that additional storage capacity is necessary when existing capacity has been exceeded (reactively).
49
+
50
+
The reactive storage decider is enabled for all policies governing data nodes and has no configuration options.
51
+
52
+
The decider relies partially on using [data tier preference](../../manage-data/lifecycle/data-tiers.md#data-tier-allocation) allocation rather than node attributes. In particular, scaling a data tier into existence (starting the first node in a tier) will result in starting a node in any data tier that is empty if not using allocation based on data tier preference. Using the [ILM migrate](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) action to migrate between tiers is the preferred way of allocating to tiers and fully supports scaling a tier into existence.
The [autoscaling](../../deploy-manage/autoscaling.md) proactive storage decider (`proactive_storage`) calculates the storage required to contain the current data set plus an estimated amount of expected additional data.
57
+
58
+
The proactive storage decider is enabled for all policies governing nodes with the `data_hot` role.
59
+
60
+
The estimation of expected additional data is based on past indexing that occurred within the `forecast_window`. Only indexing into data streams contributes to the estimate.
: (Optional, [time value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) The window of time to use for forecasting. Defaults to 30 minutes.
The [autoscaling](../../deploy-manage/autoscaling.md) frozen shards decider (`frozen_shards`) calculates the memory required to search the current set of partially mounted indices in the frozen tier. Based on a required memory amount per shard, it calculates the necessary memory in the frozen tier.
: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) The memory needed per shard, in bytes. Defaults to 2000 shards per 64 GB node (roughly 32 MB per shard). Notice that this is total memory, not heap, assuming that the Elasticsearch default heap sizing mechanism is used and that nodes are not bigger than 64 GB.
The [autoscaling](../../deploy-manage/autoscaling.md) frozen storage decider (`frozen_storage`) calculates the local storage required to search the current set of partially mounted indices based on a percentage of the total data set size of such indices. It signals that additional storage capacity is necessary when existing capacity is less than the percentage multiplied by total data set size.
104
+
105
+
The frozen storage decider is enabled for all policies governing frozen data nodes and has no configuration options.
: (Optional, number value) Percentage of local storage relative to the data set size. Defaults to 5.
111
+
112
+
## Frozen existence decider [autoscaling-frozen-existence-decider]
113
+
114
+
The [autoscaling](../../deploy-manage/autoscaling.md) frozen existence decider (`frozen_existence`) ensures that once the first index enters the frozen ILM phase, the frozen tier is scaled into existence.
115
+
116
+
The frozen existence decider is enabled for all policies governing frozen data nodes and has no configuration options.
The [autoscaling](../../deploy-manage/autoscaling.md) {{ml}} decider (`ml`) calculates the memory and CPU requirements to run {{ml}} jobs and trained models.
121
+
122
+
The {{ml}} decider is enabled for policies governing `ml` nodes.
123
+
124
+
::::{note}
125
+
For {{ml}} jobs to open when the cluster is not appropriately scaled, set `xpack.ml.max_lazy_ml_nodes` to the largest number of possible {{ml}} nodes (refer to [Advanced machine learning settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md#advanced-ml-settings) for more information). In {{ess}}, this is automatically set.
Both `num_anomaly_jobs_in_queue` and `num_analytics_jobs_in_queue` are designed to delay a scale-up event. If the cluster is too small, these settings indicate how many jobs of each type can be unassigned from a node. Both settings are only considered for jobs that can be opened given the current scale. If a job is too large for any node size or if a job can’t be assigned without user intervention (for example, a user calling `_stop` against a real-time {{anomaly-job}}), the numbers are ignored for that particular job.
132
+
133
+
`num_anomaly_jobs_in_queue`
134
+
: (Optional, integer) Specifies the number of queued {{anomaly-jobs}} to allow. Defaults to `0`.
135
+
136
+
`num_analytics_jobs_in_queue`
137
+
: (Optional, integer) Specifies the number of queued {{dfanalytics-jobs}} to allow. Defaults to `0`.
138
+
139
+
`down_scale_delay`
140
+
: (Optional, [time value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) Specifies the time to delay before scaling down. Defaults to 1 hour. If a scale down is possible for the entire time window, then a scale down is requested. If the cluster requires a scale up during the window, the window is reset.
This example creates an autoscaling policy named `my_autoscaling_policy` that overrides the default configuration of the {{ml}} decider.
146
+
147
+
```console
148
+
PUT /_autoscaling/policy/my_autoscaling_policy
149
+
{
150
+
"roles" : [ "ml" ],
151
+
"deciders": {
152
+
"ml": {
153
+
"num_anomaly_jobs_in_queue": 5,
154
+
"num_analytics_jobs_in_queue": 3,
155
+
"down_scale_delay": "30m"
156
+
}
157
+
}
158
+
}
159
+
```
160
+
161
+
The API returns the following result:
162
+
163
+
```console-result
164
+
{
165
+
"acknowledged": true
166
+
}
167
+
```
168
+
169
+
## Fixed decider [autoscaling-fixed-decider]
170
+
171
+
::::{warning}
172
+
This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
173
+
::::
174
+
175
+
176
+
::::{warning}
177
+
The fixed decider is intended for testing only. Do not use this decider in production.
178
+
::::
179
+
180
+
181
+
The [autoscaling](../../deploy-manage/autoscaling.md)`fixed` decider responds with a fixed required capacity. It is not enabled by default but can be enabled for any policy by explicitly configuring it.
0 commit comments