Skip to content

Commit 99c6abc

Browse files
authored
Merge pull request #104025 from dagiro/freshness186
freshness186
2 parents 93dbd93 + b19eb8f commit 99c6abc

File tree

1 file changed

+23
-23
lines changed

1 file changed

+23
-23
lines changed

articles/hdinsight/hdinsight-autoscale-clusters.md

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,17 @@ author: hrasheed-msft
55
ms.author: hrasheed
66
ms.reviewer: jasonh
77
ms.service: hdinsight
8-
ms.custom: hdinsightactive
98
ms.topic: conceptual
10-
ms.date: 10/22/2019
9+
ms.custom: hdinsightactive
10+
ms.date: 02/11/2020
1111
---
1212

1313
# Automatically scale Azure HDInsight clusters
1414

1515
> [!Important]
16-
> The Autoscale feature only works for Spark, Hive, LLAP and HBase clusters created after May 8th 2019.
16+
> The Autoscale feature only works for Apache Spark, ApacheHive, LLAP and Apache HBase clusters created after May 8th 2019.
1717
18-
Azure HDInsights cluster Autoscale feature automatically scales the number of worker nodes in a cluster up and down. Other types of nodes in the cluster can't be scaled currently. During the creation of a new HDInsight cluster, a minimum and maximum number of worker nodes can be set. Autoscale then monitors the resource requirements of the analytics load and scales the number of worker nodes up or down. There's no additional charge for this feature.
18+
Azure HDInsight's cluster Autoscale feature automatically scales the number of worker nodes in a cluster up and down. Other types of nodes in the cluster can't be scaled currently. During the creation of a new HDInsight cluster, a minimum and maximum number of worker nodes can be set. Autoscale then monitors the resource requirements of the analytics load and scales the number of worker nodes up or down. There's no additional charge for this feature.
1919

2020
## Cluster compatibility
2121

@@ -40,12 +40,14 @@ Schedule-based scaling changes the number of nodes in your cluster based on cond
4040

4141
Autoscale continuously monitors the cluster and collects the following metrics:
4242

43-
* **Total Pending CPU**: The total number of cores required to start execution of all pending containers.
44-
* **Total Pending Memory**: The total memory (in MB) required to start execution of all pending containers.
45-
* **Total Free CPU**: The sum of all unused cores on the active worker nodes.
46-
* **Total Free Memory**: The sum of unused memory (in MB) on the active worker nodes.
47-
* **Used Memory per Node**: The load on a worker node. A worker node on which 10 GB of memory is used, is considered under more load than a worker with 2 GB of used memory.
48-
* **Number of Application Masters per Node**: The number of Application Master (AM) containers running on a worker node. A worker node that is hosting two AM containers, is considered more important than a worker node that is hosting zero AM containers.
43+
|Metric|Description|
44+
|---|---|
45+
|Total Pending CPU|The total number of cores required to start execution of all pending containers.|
46+
|Total Pending Memory|The total memory (in MB) required to start execution of all pending containers.|
47+
|Total Free CPU|The sum of all unused cores on the active worker nodes.|
48+
|Total Free Memory|The sum of unused memory (in MB) on the active worker nodes.|
49+
|Used Memory per Node|The load on a worker node. A worker node on which 10 GB of memory is used, is considered under more load than a worker with 2 GB of used memory.|
50+
|Number of Application Masters per Node|The number of Application Master (AM) containers running on a worker node. A worker node that is hosting two AM containers, is considered more important than a worker node that is hosting zero AM containers.|
4951

5052
The above metrics are checked every 60 seconds. Autoscale makes scale-up and scale-down decisions based on these metrics.
5153

@@ -71,11 +73,9 @@ Based on the number of AM containers per node and the current CPU and memory req
7173

7274
### Create a cluster with load-based Autoscaling
7375

74-
To use Autoscale on a cluster, the **Enable autoscale** option must be enabled when the cluster is created.
75-
76-
To enable the Autoscale feature with load-based scaling, complete the following steps as part of the normal cluster creation process:
76+
To use Autoscale on a cluster, the **Enable autoscale** option must be enabled when the cluster is created. To enable the Autoscale feature with load-based scaling, complete the following steps as part of the normal cluster creation process:
7777

78-
1. On the **Configuration + pricing** tab, check the **Enable autoscale** checkbox.
78+
1. On the **Configuration + pricing** tab, select the **Enable autoscale** checkbox.
7979
1. Select **Load-based** under **Autoscale type**.
8080
1. Enter the desired values for the following properties:
8181

@@ -85,7 +85,7 @@ To enable the Autoscale feature with load-based scaling, complete the following
8585

8686
![Enable worker node load-based autoscale](./media/hdinsight-autoscale-clusters/azure-portal-cluster-configuration-pricing-autoscale.png)
8787

88-
The initial number of worker nodes must fall between the minimum and maximum, inclusive. This value defines the initial size of the cluster when it's created. The minimum number of worker nodes should be set to three or more. . Scaling your cluster to fewer than three nodes can result in it getting stuck in safe mode because of insufficient file replication. See [Getting stuck in safe mode]( https://docs.microsoft.com/ azure/hdinsight/hdinsight-scaling-best-practices#getting-stuck-in-safe-mode) for more information.
88+
The initial number of worker nodes must fall between the minimum and maximum, inclusive. This value defines the initial size of the cluster when it's created. The minimum number of worker nodes should be set to three or more. Scaling your cluster to fewer than three nodes can result in it getting stuck in safe mode because of insufficient file replication. For more information, see [Getting stuck in safe mode](./hdinsight-scaling-best-practices.md#getting-stuck-in-safe-mode).
8989

9090
### Create a cluster with schedule-based Autoscaling
9191

@@ -94,7 +94,7 @@ To enable the Autoscale feature with schedule-based scaling, complete the follow
9494
1. On the **Configuration + pricing** tab, check the **Enable autoscale** checkbox.
9595
1. Enter the **Number of nodes** for **Worker node**, which controls the limit for scaling up the cluster.
9696
1. Select the option **Schedule-based** under **Autoscale type**.
97-
1. Click **Configure** to open the **Autoscale configuration** window.
97+
1. Select **Configure** to open the **Autoscale configuration** window.
9898
1. Select your timezone and then click **+ Add condition**
9999
1. Select the days of the week that the new condition should apply to.
100100
1. Edit the time the condition should take effect and the number of nodes that the cluster should be scaled to.
@@ -185,7 +185,7 @@ You can create an HDInsight cluster with schedule-based Autoscaling an Azure Res
185185

186186
#### Using the Azure portal
187187

188-
To enable Autoscale on a running cluster, select **Cluster size** under **Settings**. Then click **Enable autoscale**. Select the type of Autoscale that you want and enter the options for load-based or schedule-based scaling. Finally, click **Save**.
188+
To enable Autoscale on a running cluster, select **Cluster size** under **Settings**. Then select **Enable autoscale**. Select the type of Autoscale that you want and enter the options for load-based or schedule-based scaling. Finally, select **Save**.
189189

190190
![Enable worker node schedule-based autoscale running cluster](./media/hdinsight-autoscale-clusters/azure-portal-settings-autoscale.png)
191191

@@ -214,7 +214,7 @@ Consider the following factors before making a decision on which mode to choose:
214214
* Enable Autoscale during cluster creation.
215215
* The minimum number of nodes should be at least three.
216216
* Load variance: does the load of the cluster follow a consistent pattern at specific times, on specific days. If not, load based scheduling is a better option.
217-
* SLA requirements: Autoscale scaling is reactive instead of predictive. Will there be a sufficient delay between when the load starts to increase and when the cluster needs to be at its target size? If there are strict SLA requirements and the load is a fixed known pattern, schedule based is a better option.
217+
* SLA requirements: Autoscale scaling is reactive instead of predictive. Will there be a sufficient delay between when the load starts to increase and when the cluster needs to be at its target size? If there are strict SLA requirements and the load is a fixed known pattern, 'schedule based' is a better option.
218218

219219
### Consider the latency of scale up or scale down operations
220220

@@ -228,7 +228,7 @@ The running jobs will continue to run and finish. The pending jobs will wait to
228228

229229
### Minimum cluster size
230230

231-
Do not scale your cluster down to fewer than three nodes. Scaling your cluster to fewer than three nodes can result in it getting stuck in safe mode because of insufficient file replication. See [Getting stuck in safe mode]( https://docs.microsoft.com/ azure/hdinsight/hdinsight-scaling-best-practices#getting-stuck-in-safe-mode) for more information.
231+
Don't scale your cluster down to fewer than three nodes. Scaling your cluster to fewer than three nodes can result in it getting stuck in safe mode because of insufficient file replication. For more information, see [Getting stuck in safe mode](./hdinsight-scaling-best-practices.md#getting-stuck-in-safe-mode).
232232

233233
## Monitoring
234234

@@ -240,24 +240,24 @@ The cluster status listed in the Azure portal can help you monitor Autoscale act
240240

241241
All of the cluster status messages that you might see are explained in the list below.
242242

243-
| Cluster status | Explanation |
243+
| Cluster status | Description |
244244
|---|---|
245245
| Running | The cluster is operating normally. All of the previous Autoscale activities have completed successfully. |
246246
| Updating | The cluster Autoscale configuration is being updated. |
247247
| HDInsight configuration | A cluster scale up or scale down operation is in progress. |
248248
| Updating Error | HDInsight encountered issues during the Autoscale configuration update. Customers can choose to either retry the update or disable autoscale. |
249249
| Error | Something is wrong with the cluster, and it isn't usable. Delete this cluster and create a new one. |
250250

251-
To view the current number of nodes in your cluster, go to the **Cluster size** chart on the **Overview** page for your cluster, or click **Cluster size** under **Settings**.
251+
To view the current number of nodes in your cluster, go to the **Cluster size** chart on the **Overview** page for your cluster, or select **Cluster size** under **Settings**.
252252

253253
### Operation history
254254

255255
You can view the cluster scale-up and scale-down history as part of the cluster metrics. You can also list all scaling actions over the past day, week, or other period of time.
256256

257-
Select **Metrics** under **Monitoring**. Then click **Add metric** and **Number of Active Workers** from the **Metric** dropdown box. Click the button in the upper right to change the time range.
257+
Select **Metrics** under **Monitoring**. Then select **Add metric** and **Number of Active Workers** from the **Metric** dropdown box. Select the button in the upper right to change the time range.
258258

259259
![Enable worker node schedule-based autoscale metric](./media/hdinsight-autoscale-clusters/hdinsight-autoscale-clusters-chart-metric.png)
260260

261261
## Next steps
262262

263-
* Read about best practices for scaling clusters manually in [Scaling best practices](hdinsight-scaling-best-practices.md)
263+
Read about best practices for scaling clusters manually in [Scaling best practices](hdinsight-scaling-best-practices.md)

0 commit comments

Comments
 (0)