Skip to content

Commit e40ea36

Browse files
authored
Merge pull request #158105 from mumian/0511-quickstarts-1
fix quickstart template broken links
2 parents a44f55d + 902d99c commit e40ea36

File tree

9 files changed

+256
-256
lines changed

9 files changed

+256
-256
lines changed

articles/hdinsight/hdinsight-autoscale-clusters.md

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ To enable the Autoscale feature with load-based scaling, complete the following
8080

8181
1. On the **Configuration + pricing** tab, select the **Enable autoscale** checkbox.
8282
1. Select **Load-based** under **Autoscale type**.
83-
1. Enter the intended values for the following properties:
83+
1. Enter the intended values for the following properties:
8484

8585
* Initial **Number of nodes** for **Worker node**.
8686
* **Min** number of worker nodes.
@@ -115,16 +115,16 @@ Select the VM type for worker nodes by selecting a VM from the drop-down list un
115115

116116
Your subscription has a capacity quota for each region. The total number of cores of your head nodes and the maximum worker nodes can't exceed the capacity quota. However, this quota is a soft limit; you can always create a support ticket to get it increased easily.
117117

118-
> [!Note]
118+
> [!Note]
119119
> If you exceed the total core quota limit, You will receive an error message saying 'the maximum node exceeded the available cores in this region, please choose another region or contact the support to increase the quota.'
120120
121-
For more information on HDInsight cluster creation using the Azure portal, see [Create Linux-based clusters in HDInsight using the Azure portal](hdinsight-hadoop-create-linux-clusters-portal.md).
121+
For more information on HDInsight cluster creation using the Azure portal, see [Create Linux-based clusters in HDInsight using the Azure portal](hdinsight-hadoop-create-linux-clusters-portal.md).
122122

123123
### Create a cluster with a Resource Manager template
124124

125125
#### Load-based autoscaling
126126

127-
You can create an HDInsight cluster with load-based Autoscaling an Azure Resource Manager template, by adding an `autoscale` node to the `computeProfile` > `workernode` section with the properties `minInstanceCount` and `maxInstanceCount` as shown in the json snippet below. For a complete Resource Manager template see [Quickstart template: Deploy Spark Cluster with load-based autoscale enabled](https://github.com/Azure/azure-quickstart-templates/tree/master/101-hdinsight-autoscale-loadbased).
127+
You can create an HDInsight cluster with load-based Autoscaling an Azure Resource Manager template, by adding an `autoscale` node to the `computeProfile` > `workernode` section with the properties `minInstanceCount` and `maxInstanceCount` as shown in the json snippet below. For a complete Resource Manager template see [Quickstart template: Deploy Spark Cluster with load-based autoscale enabled](https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.hdinsight/hdinsight-autoscale-loadbased).
128128

129129
```json
130130
{
@@ -152,7 +152,7 @@ You can create an HDInsight cluster with load-based Autoscaling an Azure Resourc
152152

153153
#### Schedule-based autoscaling
154154

155-
You can create an HDInsight cluster with schedule-based Autoscaling an Azure Resource Manager template, by adding an `autoscale` node to the `computeProfile` > `workernode` section. The `autoscale` node contains a `recurrence` that has a `timezone` and `schedule` that describes when the change will take place. For a complete Resource Manager template, see [Deploy Spark Cluster with schedule-based Autoscale Enabled](https://github.com/Azure/azure-quickstart-templates/tree/master/101-hdinsight-autoscale-schedulebased).
155+
You can create an HDInsight cluster with schedule-based Autoscaling an Azure Resource Manager template, by adding an `autoscale` node to the `computeProfile` > `workernode` section. The `autoscale` node contains a `recurrence` that has a `timezone` and `schedule` that describes when the change will take place. For a complete Resource Manager template, see [Deploy Spark Cluster with schedule-based Autoscale Enabled](https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.hdinsight/hdinsight-autoscale-schedulebased).
156156

157157
```json
158158
{
@@ -248,15 +248,15 @@ The running jobs will continue. The pending jobs will wait for scheduling with f
248248

249249
### Configure schedule-based Autoscale based on usage pattern
250250

251-
You need to understand your cluster usage pattern when you configure schedule based Autoscale. [Grafana dashboard](https://docs.microsoft.com/azure/hdinsight/interactive-query/hdinsight-grafana) can help you understand your query load and execution slots. You can get the available executor slots and total executor slots from the dashboard.
251+
You need to understand your cluster usage pattern when you configure schedule based Autoscale. [Grafana dashboard](./interactive-query/hdinsight-grafana.md) can help you understand your query load and execution slots. You can get the available executor slots and total executor slots from the dashboard.
252252

253-
Here is a way you can estimate how many worker nodes will be needed. We recommend giving additional 10% buffer to handle the variation of the workload.
253+
Here is a way you can estimate how many worker nodes will be needed. We recommend giving additional 10% buffer to handle the variation of the workload.
254254

255255
Number of executor slots actually used = Total executor slots – Total available executor slots.
256256

257257
Number of worker nodes required = Number of executor slots actually used / (hive.llap.daemon.num.executors + hive.llap.daemon.task.scheduler.wait.queue.size)
258258

259-
*hive.llap.daemon.num.executors is configurable and default is 4
259+
*hive.llap.daemon.num.executors is configurable and default is 4
260260

261261
*hive.llap.daemon.task.scheduler.wait.queue.size is configurable and default is 10
262262

@@ -267,13 +267,13 @@ Don't scale your cluster down to fewer than three nodes. Scaling your cluster to
267267

268268
### Increase the number of mappers and reducers
269269

270-
Autoscale for Hadoop clusters also monitors HDFS usage. If the HDFS is busy, it assumes the cluster still needs the current resources. When there is massive data involved in the query, you can increase the number of mappers and reducers to increase the parallelism and accelerate the HDFS operations. In this way, proper scaling down will be triggered when there are extra resources.
270+
Autoscale for Hadoop clusters also monitors HDFS usage. If the HDFS is busy, it assumes the cluster still needs the current resources. When there is massive data involved in the query, you can increase the number of mappers and reducers to increase the parallelism and accelerate the HDFS operations. In this way, proper scaling down will be triggered when there are extra resources.
271271

272272
### Set the Hive configuration Maximum Total Concurrent Queries for the peak usage scenario
273273

274274
Autoscale events don't change the Hive configuration *Maximum Total Concurrent Queries* in Ambari. This means that the Hive Server 2 Interactive Service can handle only the given number of concurrent queries at any point of time even if the Interactive Query daemons count are scaled up and down based on load and schedule. The general recommendation is to set this configuration for the peak usage scenario to avoid manual intervention.
275275

276-
However, you may experience a Hive Server 2 restart failure if there are only a small number of worker nodes and the value for maximum total concurrent queries is configured too high. At a minimum, you need the minimum number of worker nodes that can accommodate the given number of Tez Ams (equal to the Maximum Total Concurrent Queries configuration).
276+
However, you may experience a Hive Server 2 restart failure if there are only a small number of worker nodes and the value for maximum total concurrent queries is configured too high. At a minimum, you need the minimum number of worker nodes that can accommodate the given number of Tez Ams (equal to the Maximum Total Concurrent Queries configuration).
277277

278278
## Limitations
279279

0 commit comments

Comments
 (0)