Skip to content

Commit 6fe9522

Browse files
GitHubber17GitHubber17
authored
Merge pull request #86540 from hrasheed-msft/hdi_cluster_vm_config
Hdi cluster vm config
2 parents be53c78 + 703c338 commit 6fe9522

File tree

10 files changed

+84
-57
lines changed

10 files changed

+84
-57
lines changed

‎articles/hdinsight/domain-joined/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,8 @@
6666
href: ./domain-joined-zeppelin-signin.md
6767
- name: Reference
6868
items:
69+
- name: Supported node configurations
70+
href: ../hdinsight-supported-node-configuration.md
6971
- name: Code samples
7072
href: https://azure.microsoft.com/resources/samples/?service=hdinsight
7173
- name: Azure PowerShell

‎articles/hdinsight/hadoop/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,8 @@
381381
href: apache-hadoop-debug-jobs.md
382382
- name: Reference
383383
items:
384+
- name: Supported node configurations
385+
href: ../hdinsight-supported-node-configuration.md
384386
- name: Code samples
385387
href: https://azure.microsoft.com/resources/samples/?service=hdinsight
386388
- name: Azure PowerShell

‎articles/hdinsight/hbase/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,8 @@
210210
href: ../hdinsight-hadoop-optimize-hive-query.md?toc=/azure/hdinsight/hbase/toc.json&bc=/azure/hdinsight/hbase/breadcrumb/toc.json
211211
- name: Reference
212212
items:
213+
- name: Supported node configurations
214+
href: ../hdinsight-supported-node-configuration.md
213215
- name: Code samples
214216
href: https://azure.microsoft.com/resources/samples/?service=hdinsight
215217
- name: Azure PowerShell

‎articles/hdinsight/hdinsight-component-versioning.md

Lines changed: 1 addition & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -172,63 +172,7 @@ The section provides links to release notes for the Hortonworks Data Platform di
172172

173173
## Default node configuration and virtual machine sizes for clusters
174174

175-
The following tables list the default virtual machine (VM) sizes for HDInsight clusters. This chart is necessary to understand the VM sizes to use when you are creating PowerShell or Azure CLI scripts to deploy HDInsight clusters.
176-
177-
> [!NOTE]
178-
> The only cluster types that have data disks are Kafka and HBase clusters with the Accelerated Writes feature enabled. HDInsight supports P30 and S30 disk sizes in these scenarios.
179-
180-
> [!IMPORTANT]
181-
> If you need more than 32 worker nodes in a cluster, you must select a head node size with at least 8 cores and 14 GB of RAM.
182-
183-
* All supported regions except Brazil South and Japan West:
184-
185-
|Cluster type|Hadoop|HBase|Interactive Query|Storm|Spark|ML Server|Kafka|
186-
|---|---|---|---|---|---|---|---|
187-
|Head: default VM size|D12 v2|D12 v2|D13 v2|A3|D12 v2|D12 v2|D3v2|
188-
|Head: recommended VM sizes|D3 v2|D3 v2|D13|A4 v2|D12 v2|D12 v2|A2M v2|
189-
||D4 v2|D4 v2|D14|A8 v2|D13 v2|D13 v2|D3 v2|
190-
||D12 v2|D12 v2|E16 v3|A2m v2|D14 v2|D14 v2|D4 v2|
191-
||E4 v3|E4 v3|E32 v3|E4 v3|E4 v3|E4 v3|D12 v2|
192-
|Worker: default VM size|D4 v2|D4 v2|D14 v2|D3 v2|D13 v2|D4 v2|4 D12v2 with 2 S30 disks per broker|
193-
|Worker: recommended VM sizes|D3 v2|D3 v2|D13|D3 v2|D4 v2|D4 v2|D13 v2|
194-
||D4 v2|D4 v2|D14|D4 v2|D12 v2|D12 v2|DS12 v2|
195-
||D12 v2|D12 v2|E16 v3|D12 v2|D13 v2|D13 v2|DS13 v2|
196-
||E4 v3|E4 v3|E20 v3|E4 v3|D14 v2|D14 v2|E4 v3|
197-
||||E32 v3||E16 v3|E16 v3|ES4 v3|
198-
||||E64 v3||E20 v3|E20 v3|E8 v3|
199-
||||||E32 v3|E32 v3|ES8 v3|
200-
||||||E64 v3|E64 v3||
201-
|ZooKeeper: default VM size||A4 v2|A4 v2|A4 v2||A2 v2|D3v2|
202-
|ZooKeeper: recommended VM sizes||A4 v2||A2 v2|||A2M v2|
203-
|||A8 v2||A4 v2|||D3 v2|
204-
|||A2m v2||A8 v2|||E8 v3|
205-
|ML Services: default VM size||||||D4 v2||
206-
|ML Services: recommended VM size||||||D4 v2||
207-
|||||||D12 v2||
208-
|||||||D13 v2||
209-
|||||||D14 v2||
210-
|||||||E16 v3||
211-
|||||||E20 v3||
212-
|||||||E32 v3||
213-
|||||||E64 v3||
214-
215-
* Brazil South and Japan West only (no v2 sizes):
216-
217-
| Cluster type | Hadoop | HBase | Interactive Query |Storm | Spark | ML Services |
218-
| --- | --- | --- | --- | --- | --- | --- |
219-
| Head: default VM size |D12 |D12 | D13 |A3 |D12 |D12 |
220-
| Head: recommended VM sizes |D3,<br/> D4,<br/> D12 |D3,<br/> D4,<br/> D12 | D13,<br/> D14 |A3,<br/> A4,<br/> A5 |D12,<br/> D13,<br/> D14 |D12,<br/> D13,<br/> D14 |
221-
| Worker: default VM size |D4 |D4 | D14 |D3 |D13 |D4 |
222-
| Worker: recommended VM sizes |D3,<br/> D4,<br/> D12 |D3,<br/> D4,<br/> D12 | D13,<br/> D14 |D3,<br/> D4,<br/> D12 |D4,<br/> D12,<br/> D13,<br/> D14 | D4,<br/> D12,<br/> D13,<br/> D14 |
223-
| ZooKeeper: default VM size | |A4 v2 | A4 v2| A4 v2 | | A2 v2|
224-
| ZooKeeper: recommended VM sizes | |A2,<br/> A3,<br/> A4 | |A2,<br/> A3,<br/> A4 | | |
225-
| ML Services: default VM sizes | | | | | |D4 |
226-
| ML Services: recommended VM sizes | | | | | |D4,<br/> D12,<br/> D13,<br/> D14 |
227-
228-
> [!NOTE]
229-
> - Head is known as *Nimbus* for the Storm cluster type.
230-
> - Worker is known as *Supervisor* for the Storm cluster type.
231-
> - Worker is known as *Region* for the HBase cluster type.
175+
For more information on which virtual machine SKUs to select for your cluster, see [Azure HDInsight cluster configuration details](hdinsight-supported-node-configuration.md).
232176

233177
## Next steps
234178
- [Cluster setup for Apache Hadoop, Spark, and more on HDInsight](hdinsight-hadoop-provision-linux-clusters.md)

‎articles/hdinsight/hdinsight-supported-node-configuration.md

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
---
2+
title: Azure HDInsight supported node configurations
3+
description: Learn the minimum and recommended configurations for HDInsight cluster nodes.
4+
keywords: vm sizes, cluster sizes, cluster configuration
5+
author: hrasheed-msft
6+
ms.author: hrasheed
7+
ms.reviewer: jasonh
8+
ms.service: hdinsight
9+
ms.custom: hdinsightactive,hdiseo17may2017
10+
ms.topic: conceptual
11+
ms.date: 08/26/2019
12+
---
13+
# What are the default and recommended node configurations for Azure HDInsight?
14+
15+
This article discusses default and recommended node configurations for Azure HDInsight clusters.
16+
17+
## Default and recommended node configuration and virtual machine sizes for clusters
18+
19+
The following tables list default and recommended virtual machine (VM) sizes for HDInsight clusters. This information is necessary to understand the VM sizes to use when you are creating PowerShell or Azure CLI scripts to deploy HDInsight clusters.
20+
21+
If you need more than 32 worker nodes in a cluster, select a head node size with at least 8 cores and 14 GB of RAM.
22+
23+
The only cluster types that have data disks are Kafka and HBase clusters with the Accelerated Writes feature enabled. HDInsight supports P30 and S30 disk sizes in these scenarios.
24+
25+
For more details on the specification of each VM type, see the following documents:
26+
27+
* [General purpose virtual machine sizes: Dv2 series 1-5](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-general#dv2-series)
28+
* [Memory optimized virtual machine sizes: Dv2 series 11-15](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-memory#dv2-series-11-15)
29+
* [General purpose virtual machine sizes: Av2 series 1-8](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-general#av2-series)
30+
31+
### All supported regions except Brazil south and Japan west
32+
33+
> [!Note]
34+
> To get the SKU identifier for use in powershell and other scripts, add `Standard_` to the beginning of all of the VM SKUs in the tables below. For example, `D12_v2` would become `Standard_D12_v2`.
35+
36+
| Cluster type | Hadoop | HBase | Interactive Query | Storm | Spark | ML Server | Kafka |
37+
|---|---|---|---|---|---|---|---|
38+
| Head: default VM size | D12_v2 | D12_v2 | D13_v2 | A4_v2 | D12_v2 | D12_v2 | D3_v2 |
39+
| Head: recommended VM sizes | D13_v2,<br/>D14_v2,<br/>D5_v2 | D3_v2,<br/>D4_v2,<br/>D12_v2 | D13_v2,<br/>D14_v2 | A4_v2,<br/>A8_v2 | D12_v2,<br/>D13_v2,<br/>D14_v2 | D12_v2,<br/>D13_v2,<br/>D14_v2 | D3_v2,<br/>D4_v2,<br/>D12_v2 |
40+
| Worker: default VM size | D4_v2 | D4_v2 | D14_v2 | D3_v2 | D13_v2 | D4_v2 | 4 D12_v2 with 2 S30 disks per broker |
41+
| Worker: recommended VM sizes | D5_v2,<br>D12_v2,<br/>D13_v2 | D3_v2,<br/>D4_v2,<br/>D13_v2 | D13_v2,<br/>D14_v2 | D3_v2<br/>D4_v2,<br/>D12_v2 | D12_v2,<br>D13_v2,<br>D14_v2 | D4_v2,<br/>D12_v2,<br>D13_v2,<br>D14_v2 | D3_v2,<br/>D4_v2,<br/>DS3_v2,<br/>DS4_v2 |
42+
| ZooKeeper: default VM size | | A4_v2 | A4_v2 | A4_v2 | | A2_v2 | A4_v2 |
43+
| ZooKeeper: recommended VM sizes | | A4_v2, <br/>A8_v2, <br/>A2m_v2 | A4_v2,<br/>A8_v2,<br/>A2m_v2 | A4_v2,<br/>A2_v2,<br/>A8_v2 | | A2_v2 | A4_v2,<br/> A8_v2,<br/>A2m_v2 |
44+
| ML Services: default VM size | | | | | | D4_v2 | |
45+
| ML Services: recommended VM size | | | | | | D4_v2,<br/> D12_v2,<br/> D13_v2,<br/>D14_v2 | |
46+
47+
### Brazil south and Japan west only
48+
49+
| Cluster type | Hadoop | HBase | Interactive Query | Storm | Spark | ML Services |
50+
|---|---|---|---|---|---|---|
51+
| Head: default VM size | D12 | D12 | D13 | A4_v2 | D12 | D12 |
52+
| Head: recommended VM sizes | D5_v2,<br/> D13_v2,<br/> D14_v2 | D3_v2,<br/> D4_v2,<br/> D12_v2 | D13_v2,<br/> D14_v2 | A4_v2,<br/> A8_v2 | D12_v2,<br/> D13_v2,<br/> D14_v2 | D12_v2,<br/> D13_v2,<br/> D14_v2 |
53+
| Worker: default VM size | D4 | D4 | D14 | D3 | D13 | D4 |
54+
| Worker: recommended VM sizes | D5_v2,<br/> D12_v2,<br/> D13_v2 | D3_v2,<br/> D4_v2,<br/> D13_v2 | D13_v2,<br/> D14_v2 | D3_v2,<br/> D4_v2,<br/> D12_v2 | D12_v2,<br/> D13_v2,<br/> D14_v2 | D4_v2,<br/> D12_v2,<br/> D13_v2,<br/> D14_v2 |
55+
| ZooKeeper: default VM size | | A4_v2 | A4_v2 | A4_v2 | | A2_v2 |
56+
| ZooKeeper: recommended VM sizes | | A4_v2,<br/> A8_v2,<br/> A2m_v2 | A4_v2,<br/> A8_v2,<br/> A2m_v2 | A4_v2,<br/> A8_v2 | | A2_v2 |
57+
| ML Services: default VM sizes | | | | | | D4 |
58+
| ML Services: recommended VM sizes | | | | | | D4_v2,<br/> D12_v2,<br/> D13_v2,<br/> D14_v2 |
59+
60+
> [!NOTE]
61+
> - Head is known as *Nimbus* for the Storm cluster type.
62+
> - Worker is known as *Supervisor* for the Storm cluster type.
63+
> - Worker is known as *Region* for the HBase cluster type.
64+
65+
## Next steps
66+
67+
* [What are the Apache Hadoop components and versions available with HDInsight?](hdinsight-component-versioning.md)

‎articles/hdinsight/interactive-query/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -281,6 +281,8 @@
281281
maintainContext: true
282282
- name: Reference
283283
items:
284+
- name: Supported node configurations
285+
href: ../hdinsight-supported-node-configuration.md
284286
- name: Code samples
285287
href: https://azure.microsoft.com/resources/samples/?service=hdinsight
286288
- name: Azure PowerShell

‎articles/hdinsight/kafka/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,8 @@
181181
href: ../hdinsight-hadoop-collect-debug-heap-dump-linux.md?toc=/azure/hdinsight/kafka/toc.json&bc=/azure/hdinsight/kafka/breadcrumb/toc.json
182182
- name: Reference
183183
items:
184+
- name: Supported node configurations
185+
href: ../hdinsight-supported-node-configuration.md
184186
- name: Code samples
185187
href: https://azure.microsoft.com/resources/samples/?service=hdinsight
186188
- name: Azure PowerShell

‎articles/hdinsight/r-server/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,8 @@
203203
href: ../hdinsight-migrate-granular-access-cluster-configurations.md?toc=/azure/hdinsight/r-server/toc.json&bc=/azure/hdinsight/r-server/breadcrumb/toc.json
204204
- name: Reference
205205
items:
206+
- name: Supported node configurations
207+
href: ../hdinsight-supported-node-configuration.md
206208
- name: Code samples
207209
href: https://azure.microsoft.com/resources/samples/?service=hdinsight
208210
- name: Azure PowerShell

‎articles/hdinsight/spark/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -331,6 +331,8 @@
331331
href: ./zookeeper-troubleshoot-quorum-fails.md
332332
- name: Reference
333333
items:
334+
- name: Supported node configurations
335+
href: ../hdinsight-supported-node-configuration.md
334336
- name: Code samples
335337
href: https://azure.microsoft.com/resources/samples/?service=hdinsight
336338
- name: Azure PowerShell

‎articles/hdinsight/storm/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,8 @@
171171
href: ../hdinsight-hadoop-collect-debug-heap-dump-linux.md?toc=/azure/hdinsight/storm/toc.json&bc=/azure/hdinsight/storm/breadcrumb/toc.json
172172
- name: Reference
173173
items:
174+
- name: Supported node configurations
175+
href: ../hdinsight-supported-node-configuration.md
174176
- name: Code samples
175177
href: https://azure.microsoft.com/resources/samples/?service=hdinsight
176178
- name: Azure PowerShell

0 commit comments

Comments
 (0)