You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-autoscale-clusters.md
+12-18Lines changed: 12 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,16 +7,15 @@ ms.reviewer: jasonh
7
7
ms.service: hdinsight
8
8
ms.topic: conceptual
9
9
ms.custom: hdinsightactive
10
-
ms.date: 02/21/2020
10
+
ms.date: 03/05/2020
11
11
---
12
12
13
13
# Automatically scale Azure HDInsight clusters
14
14
15
15
> [!Important]
16
-
> The Azure HDInsight Autoscale feature was released for general availability on November 7th, 2019 for Spark and Hadoop clusters and included improvements not available in the preview version of the feature. If you created a Spark cluster prior to November 7th, 2019 and want to use the Autoscale feature on your cluster, the recommended path is to create a new cluster, and enable Autoscale on the new cluster.
16
+
> The Azure HDInsight Autoscale feature was released for general availability on November 7th, 2019 for Spark and Hadoop clusters and included improvements not available in the preview version of the feature. If you created a Spark cluster prior to November 7th, 2019 and want to use the Autoscale feature on your cluster, the recommended path is to create a new cluster, and enable Autoscale on the new cluster.
17
17
>
18
-
>Autoscale for Interactive Query (LLAP) and HBase clusters is still in preview. Autoscale is only available on Spark, Hadoop, Interactive Query, and HBase clusters.
19
-
18
+
> Autoscale for Interactive Query (LLAP) and HBase clusters is still in preview. Autoscale is only available on Spark, Hadoop, Interactive Query, and HBase clusters.
20
19
21
20
Azure HDInsight's cluster Autoscale feature automatically scales the number of worker nodes in a cluster up and down. Other types of nodes in the cluster can't be scaled currently. During the creation of a new HDInsight cluster, a minimum and maximum number of worker nodes can be set. Autoscale then monitors the resource requirements of the analytics load and scales the number of worker nodes up or down. There's no additional charge for this feature.
22
21
@@ -54,23 +53,18 @@ Autoscale continuously monitors the cluster and collects the following metrics:
54
53
55
54
The above metrics are checked every 60 seconds. Autoscale makes scale-up and scale-down decisions based on these metrics.
56
55
57
-
### Load-based cluster scale-up
58
-
59
-
When the following conditions are detected, Autoscale will issue a scale-up request:
60
-
61
-
* Total pending CPU is greater than total free CPU for more than 3 minutes.
62
-
* Total pending memory is greater than total free memory for more than 3 minutes.
56
+
### Load-based scale conditions
63
57
64
-
The HDInsight service calculates how many new worker nodes are needed to meet the current CPU and memory requirements, and then issues a scale-up request to add the required number of nodes.
58
+
When the following conditions are detected, Autoscale will issue a scale request:
65
59
66
-
### Load-based cluster scale-down
67
-
68
-
When the following conditions are detected, Autoscale will issue a scale-down request:
60
+
|Scale-up|Scale-down|
61
+
|---|---|
62
+
|Total pending CPU is greater than total free CPU for more than 3 minutes.|Total pending CPU is less than total free CPU for more than 10 minutes.|
63
+
|Total pending memory is greater than total free memory for more than 3 minutes.|Total pending memory is less than total free memory for more than 10 minutes.|
69
64
70
-
* Total pending CPU is less than total free CPU for more than 10 minutes.
71
-
* Total pending memory is less than total free memory for more than 10 minutes.
65
+
For scale-up, the HDInsight service calculates how many new worker nodes are needed to meet the current CPU and memory requirements, and then issues a scale-up request to add the required number of nodes.
72
66
73
-
Based on the number of AM containers per node and the current CPU and memory requirements, Autoscale issues a request to remove a certain number of nodes. The service also detects which nodes are candidates for removal based on current job execution. The scale down operation first decommissions the nodes, and then removes them from the cluster.
67
+
For scale-down, based on the number of AM containers per node and the current CPU and memory requirements, Autoscale issues a request to remove a certain number of nodes. The service also detects which nodes are candidates for removal based on current job execution. The scale down operation first decommissions the nodes, and then removes them from the cluster.
74
68
75
69
## Get started
76
70
@@ -113,7 +107,7 @@ For both load-based and schedule-based scaling, select the VM type for worker no
Your subscription has a capacity quota for each region. The total number of cores of your head nodes combined with the maximum number of worker nodes can’t exceed the capacity quota. However, this quota is a soft limit; you can always create a support ticket to get it increased easily.
110
+
Your subscription has a capacity quota for each region. The total number of cores of your head nodes combined with the maximum number of worker nodes can't exceed the capacity quota. However, this quota is a soft limit; you can always create a support ticket to get it increased easily.
117
111
118
112
> [!Note]
119
113
> If you exceed the total core quota limit, You will receive an error message saying 'the maximum node exceeded the available cores in this region, please choose another region or contact the support to increase the quota.'
0 commit comments