You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-autoscale-clusters.md
+23-23Lines changed: 23 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,17 +5,17 @@ author: hrasheed-msft
5
5
ms.author: hrasheed
6
6
ms.reviewer: jasonh
7
7
ms.service: hdinsight
8
-
ms.custom: hdinsightactive
9
8
ms.topic: conceptual
10
-
ms.date: 10/22/2019
9
+
ms.custom: hdinsightactive
10
+
ms.date: 02/11/2020
11
11
---
12
12
13
13
# Automatically scale Azure HDInsight clusters
14
14
15
15
> [!Important]
16
-
> The Autoscale feature only works for Spark, Hive, LLAP and HBase clusters created after May 8th 2019.
16
+
> The Autoscale feature only works for Apache Spark, ApacheHive, LLAP and Apache HBase clusters created after May 8th 2019.
17
17
18
-
Azure HDInsight’s cluster Autoscale feature automatically scales the number of worker nodes in a cluster up and down. Other types of nodes in the cluster can't be scaled currently. During the creation of a new HDInsight cluster, a minimum and maximum number of worker nodes can be set. Autoscale then monitors the resource requirements of the analytics load and scales the number of worker nodes up or down. There's no additional charge for this feature.
18
+
Azure HDInsight's cluster Autoscale feature automatically scales the number of worker nodes in a cluster up and down. Other types of nodes in the cluster can't be scaled currently. During the creation of a new HDInsight cluster, a minimum and maximum number of worker nodes can be set. Autoscale then monitors the resource requirements of the analytics load and scales the number of worker nodes up or down. There's no additional charge for this feature.
19
19
20
20
## Cluster compatibility
21
21
@@ -40,12 +40,14 @@ Schedule-based scaling changes the number of nodes in your cluster based on cond
40
40
41
41
Autoscale continuously monitors the cluster and collects the following metrics:
42
42
43
-
***Total Pending CPU**: The total number of cores required to start execution of all pending containers.
44
-
***Total Pending Memory**: The total memory (in MB) required to start execution of all pending containers.
45
-
***Total Free CPU**: The sum of all unused cores on the active worker nodes.
46
-
***Total Free Memory**: The sum of unused memory (in MB) on the active worker nodes.
47
-
***Used Memory per Node**: The load on a worker node. A worker node on which 10 GB of memory is used, is considered under more load than a worker with 2 GB of used memory.
48
-
***Number of Application Masters per Node**: The number of Application Master (AM) containers running on a worker node. A worker node that is hosting two AM containers, is considered more important than a worker node that is hosting zero AM containers.
43
+
|Metric|Description|
44
+
|---|---|
45
+
|Total Pending CPU|The total number of cores required to start execution of all pending containers.|
46
+
|Total Pending Memory|The total memory (in MB) required to start execution of all pending containers.|
47
+
|Total Free CPU|The sum of all unused cores on the active worker nodes.|
48
+
|Total Free Memory|The sum of unused memory (in MB) on the active worker nodes.|
49
+
|Used Memory per Node|The load on a worker node. A worker node on which 10 GB of memory is used, is considered under more load than a worker with 2 GB of used memory.|
50
+
|Number of Application Masters per Node|The number of Application Master (AM) containers running on a worker node. A worker node that is hosting two AM containers, is considered more important than a worker node that is hosting zero AM containers.|
49
51
50
52
The above metrics are checked every 60 seconds. Autoscale makes scale-up and scale-down decisions based on these metrics.
51
53
@@ -71,11 +73,9 @@ Based on the number of AM containers per node and the current CPU and memory req
71
73
72
74
### Create a cluster with load-based Autoscaling
73
75
74
-
To use Autoscale on a cluster, the **Enable autoscale** option must be enabled when the cluster is created.
75
-
76
-
To enable the Autoscale feature with load-based scaling, complete the following steps as part of the normal cluster creation process:
76
+
To use Autoscale on a cluster, the **Enable autoscale** option must be enabled when the cluster is created. To enable the Autoscale feature with load-based scaling, complete the following steps as part of the normal cluster creation process:
77
77
78
-
1. On the **Configuration + pricing** tab, check the **Enable autoscale** checkbox.
78
+
1. On the **Configuration + pricing** tab, select the **Enable autoscale** checkbox.
79
79
1. Select **Load-based** under **Autoscale type**.
80
80
1. Enter the desired values for the following properties:
81
81
@@ -85,7 +85,7 @@ To enable the Autoscale feature with load-based scaling, complete the following
The initial number of worker nodes must fall between the minimum and maximum, inclusive. This value defines the initial size of the cluster when it's created. The minimum number of worker nodes should be set to three or more. . Scaling your cluster to fewer than three nodes can result in it getting stuck in safe mode because of insufficient file replication. See [Getting stuck in safe mode](https://docs.microsoft.com/ azure/hdinsight/hdinsight-scaling-best-practices#getting-stuck-in-safe-mode) for more information.
88
+
The initial number of worker nodes must fall between the minimum and maximum, inclusive. This value defines the initial size of the cluster when it's created. The minimum number of worker nodes should be set to three or more. Scaling your cluster to fewer than three nodes can result in it getting stuck in safe mode because of insufficient file replication. For more information, see [Getting stuck in safe mode](./hdinsight-scaling-best-practices.md#getting-stuck-in-safe-mode).
89
89
90
90
### Create a cluster with schedule-based Autoscaling
91
91
@@ -94,7 +94,7 @@ To enable the Autoscale feature with schedule-based scaling, complete the follow
94
94
1. On the **Configuration + pricing** tab, check the **Enable autoscale** checkbox.
95
95
1. Enter the **Number of nodes** for **Worker node**, which controls the limit for scaling up the cluster.
96
96
1. Select the option **Schedule-based** under **Autoscale type**.
97
-
1.Click**Configure** to open the **Autoscale configuration** window.
97
+
1.Select**Configure** to open the **Autoscale configuration** window.
98
98
1. Select your timezone and then click **+ Add condition**
99
99
1. Select the days of the week that the new condition should apply to.
100
100
1. Edit the time the condition should take effect and the number of nodes that the cluster should be scaled to.
@@ -185,7 +185,7 @@ You can create an HDInsight cluster with schedule-based Autoscaling an Azure Res
185
185
186
186
#### Using the Azure portal
187
187
188
-
To enable Autoscale on a running cluster, select **Cluster size** under **Settings**. Then click**Enable autoscale**. Select the type of Autoscale that you want and enter the options for load-based or schedule-based scaling. Finally, click**Save**.
188
+
To enable Autoscale on a running cluster, select **Cluster size** under **Settings**. Then select**Enable autoscale**. Select the type of Autoscale that you want and enter the options for load-based or schedule-based scaling. Finally, select**Save**.
@@ -214,7 +214,7 @@ Consider the following factors before making a decision on which mode to choose:
214
214
* Enable Autoscale during cluster creation.
215
215
* The minimum number of nodes should be at least three.
216
216
* Load variance: does the load of the cluster follow a consistent pattern at specific times, on specific days. If not, load based scheduling is a better option.
217
-
* SLA requirements: Autoscale scaling is reactive instead of predictive. Will there be a sufficient delay between when the load starts to increase and when the cluster needs to be at its target size? If there are strict SLA requirements and the load is a fixed known pattern, ‘schedule based’ is a better option.
217
+
* SLA requirements: Autoscale scaling is reactive instead of predictive. Will there be a sufficient delay between when the load starts to increase and when the cluster needs to be at its target size? If there are strict SLA requirements and the load is a fixed known pattern, 'schedule based' is a better option.
218
218
219
219
### Consider the latency of scale up or scale down operations
220
220
@@ -228,7 +228,7 @@ The running jobs will continue to run and finish. The pending jobs will wait to
228
228
229
229
### Minimum cluster size
230
230
231
-
Do not scale your cluster down to fewer than three nodes. Scaling your cluster to fewer than three nodes can result in it getting stuck in safe mode because of insufficient file replication. See [Getting stuck in safe mode](https://docs.microsoft.com/ azure/hdinsight/hdinsight-scaling-best-practices#getting-stuck-in-safe-mode) for more information.
231
+
Don't scale your cluster down to fewer than three nodes. Scaling your cluster to fewer than three nodes can result in it getting stuck in safe mode because of insufficient file replication. For more information, see [Getting stuck in safe mode](./hdinsight-scaling-best-practices.md#getting-stuck-in-safe-mode).
232
232
233
233
## Monitoring
234
234
@@ -240,24 +240,24 @@ The cluster status listed in the Azure portal can help you monitor Autoscale act
240
240
241
241
All of the cluster status messages that you might see are explained in the list below.
242
242
243
-
| Cluster status |Explanation|
243
+
| Cluster status |Description|
244
244
|---|---|
245
245
| Running | The cluster is operating normally. All of the previous Autoscale activities have completed successfully. |
246
246
| Updating | The cluster Autoscale configuration is being updated. |
247
247
| HDInsight configuration | A cluster scale up or scale down operation is in progress. |
248
248
| Updating Error | HDInsight encountered issues during the Autoscale configuration update. Customers can choose to either retry the update or disable autoscale. |
249
249
| Error | Something is wrong with the cluster, and it isn't usable. Delete this cluster and create a new one. |
250
250
251
-
To view the current number of nodes in your cluster, go to the **Cluster size** chart on the **Overview** page for your cluster, or click**Cluster size** under **Settings**.
251
+
To view the current number of nodes in your cluster, go to the **Cluster size** chart on the **Overview** page for your cluster, or select**Cluster size** under **Settings**.
252
252
253
253
### Operation history
254
254
255
255
You can view the cluster scale-up and scale-down history as part of the cluster metrics. You can also list all scaling actions over the past day, week, or other period of time.
256
256
257
-
Select **Metrics** under **Monitoring**. Then click**Add metric** and **Number of Active Workers** from the **Metric** dropdown box. Click the button in the upper right to change the time range.
257
+
Select **Metrics** under **Monitoring**. Then select**Add metric** and **Number of Active Workers** from the **Metric** dropdown box. Select the button in the upper right to change the time range.
0 commit comments