|
| 1 | +--- |
| 2 | +title: Reliability in Azure HDInsight |
| 3 | +description: Find out about reliability in Azure HDInsight |
| 4 | +author: apurbasroy |
| 5 | +ms.service: azure |
| 6 | +ms.topic: conceptual |
| 7 | +ms.date: 02/27/2023 |
| 8 | +ms.author: anaharris |
| 9 | +ms.custom: references_regions, subject-reliability |
| 10 | +CustomerIntent: As a cloud architect/engineer, I need general guidance on migrating HDInsight to using availability zones. |
| 11 | +--- |
| 12 | + |
| 13 | +# Reliability in Azure HDInsight |
| 14 | + |
| 15 | +This article describes reliability support in Azure HDInsight, and covers [availability zones](#availability-zone-support). For a more detailed overview of reliability in Azure, see [Azure reliability](/azure/architecture/framework/resiliency/overview). |
| 16 | + |
| 17 | + |
| 18 | +## Availability zone support |
| 19 | + |
| 20 | +Azure availability zones are at least three physically separate groups of datacenters within each Azure region. Datacenters within each zone are equipped with independent power, cooling, and networking infrastructure. In the case of a local zone failure, availability zones are designed so that if one zone is affected, regional services, capacity, and high availability are supported by the remaining two zones. Failures can range from software and hardware failures to events such as earthquakes, floods, and fires. Tolerance to failures is achieved with redundancy and logical isolation of Azure services. For more detailed information on availability zones in Azure, see [Availability zone service and regional support](availability-zones-service-support.md). |
| 21 | + |
| 22 | +Azure HDInsight supports a [zonal deployment configuration](availability-zones-service-support.md#azure-services-with-availability-zone-support). Azure HDInsight cluster nodes are placed in a single zone that you select in the selected region. A zonal HDInsight cluster is isolated from any outages that occur in other zones. However, if an outage impacts the specific zone chosen for the HDInsight cluster, the cluster won't be available. This deployment model provides inexpensive, low latency network connectivity within the cluster. Replicating this deployment model into multiple availability zones can provide a higher level of availability to protect against hardware failure. |
| 23 | + |
| 24 | +>[!IMPORTANT] |
| 25 | +>For deployments where users don't specify a specific zone, node types are not zone resilient and can experience downtime during an outage in any zone in that region. |
| 26 | +
|
| 27 | +## Prerequisites |
| 28 | + |
| 29 | +- Availability zones are only supported for clusters created after June 15, 2023. Availability zone settings can't be updated after the cluster is created. You also can't update an existing, non-availability zone cluster to use availability zones. |
| 30 | + |
| 31 | +- Clusters must be created under a custom VNet. |
| 32 | + |
| 33 | +- You need to bring your own SQL DB for Ambari DB and external metastore, such as Hive metastore, so that you can config these DBs in the same availability zone. |
| 34 | + |
| 35 | +- Your HDInsight clusters must be created with the availability zone option in one of the following regions: |
| 36 | + |
| 37 | + - Australia East |
| 38 | + - Brazil South |
| 39 | + - Canada Central |
| 40 | + - Central US |
| 41 | + - East US |
| 42 | + - East US 2 |
| 43 | + - France Central |
| 44 | + - Germany West Central |
| 45 | + - Japan East |
| 46 | + - Korea Central |
| 47 | + - North Europe |
| 48 | + - Qatar Central |
| 49 | + - Southeast Asia |
| 50 | + - South Central US |
| 51 | + - UK South |
| 52 | + - US Gov Virginia |
| 53 | + - West Europe |
| 54 | + - West US 2 |
| 55 | + |
| 56 | + |
| 57 | +## Create an HDInsight cluster using availability zone |
| 58 | + |
| 59 | +You can use Azure Resource Manager (ARM) template to launch an HDInsight cluster into a specified availability zone. |
| 60 | + |
| 61 | +In the resources section, you need to add a section of ‘zones’ and provide which availability zone you want this cluster to be deployed into. |
| 62 | + |
| 63 | +```json |
| 64 | + "resources": [ |
| 65 | + { |
| 66 | + "type": "Microsoft.HDInsight/clusters", |
| 67 | + "apiVersion": "2021-06-01", |
| 68 | + "name": "[parameters('cluster name')]", |
| 69 | + "location": "East US 2", |
| 70 | + "zones": [ |
| 71 | + "1" |
| 72 | + ], |
| 73 | + } |
| 74 | + ] |
| 75 | +``` |
| 76 | + |
| 77 | +### Verify nodes within one availability Zone across zones |
| 78 | + |
| 79 | +When the HDInsight cluster is ready, you can check the location to see which availability zone they're deployed in. |
| 80 | + |
| 81 | +:::image type="content" source="../hdinsight/media/hdinsight-use-availability-zones/cluster-availability-zone-info.png" alt-text="Screenshot that shows availability zone info in cluster overview." border="true"::: |
| 82 | + |
| 83 | +**Get API response**: |
| 84 | + |
| 85 | +```json |
| 86 | + [ |
| 87 | + { |
| 88 | + "location": "East US 2", |
| 89 | + "zones": [ |
| 90 | + "1" |
| 91 | + ], |
| 92 | + } |
| 93 | + ] |
| 94 | +``` |
| 95 | + |
| 96 | +### Scale up the cluster |
| 97 | + |
| 98 | +You can scale up an HDInsight cluster with more worker nodes. The newly added worker nodes will be placed in the same availability zone of this cluster. |
| 99 | + |
| 100 | +### Availability zone redeployment |
| 101 | + |
| 102 | +Azure HDInsight clusters currently doesn't support in-place migration of existing cluster instances to availability zone support. However, you can choose to [recreate your cluster](#create-an-hdinsight-cluster-using-availability-zone), and choose a different availability zone or region during the cluster creation. A secondary standby cluster in a different region and a different availability zone can be used in disaster recovery scenarios. |
| 103 | + |
| 104 | +### Zone down experience |
| 105 | + |
| 106 | +When an availability zone goes down: |
| 107 | + |
| 108 | + - You can't ssh to this cluster. |
| 109 | + - You can't delete or scale up or scale down this cluster. |
| 110 | + - You can't submit jobs or see job history. |
| 111 | + - You still can submit new cluster creation request in a different region. |
| 112 | + |
| 113 | + |
| 114 | +## Next steps |
| 115 | + |
| 116 | +> [!div class="nextstepaction"] |
| 117 | +> [Reliability in Azure](availability-zones-overview.md) |
0 commit comments