|
1 | 1 | ---
|
2 |
| -title: Load balancer migration guidelines for Azure HDInsight |
3 |
| -description: Guidelines to perform load balancer migration for Azure HDInsight. |
| 2 | +title: Basic Load balancer deprecation - Guidelines for Azure HDInsight |
| 3 | +description: Guidelines to transition to standard load balancer for Azure HDInsight. |
4 | 4 | ms.service: azure-hdinsight
|
5 | 5 | ms.topic: how-to
|
6 | 6 | ms.date: 09/25/2024
|
7 | 7 | ---
|
8 | 8 |
|
9 |
| -# Load balancer migration guidelines |
| 9 | +# Basic Load balancer deprecation: Guidelines for Azure HDInsight |
10 | 10 |
|
11 | 11 | This article describes the details about the impact on HDInsight clusters and the necessary steps required as HDInsight service is transitioning to use standard load balancers for all its cluster configurations.
|
12 | 12 |
|
13 | 13 | This transition is done in line with the announcement of retirement of Azure basic load balancer by 30 September 2025 and no support for new deployment by Mar 31, 2025. For more information, see [Azure Basic Load Balancer retires on 30 September 2025. Upgrade to Standard Load Balancer](https://azure.microsoft.com/updates/azure-basic-load-balancer-will-be-retired-on-30-september-2025-upgrade-to-standard-load-balancer).
|
14 | 14 |
|
15 | 15 | ## Impact on HDInsight clusters
|
16 | 16 |
|
17 |
| -### Scenario 1: Long running clusters (when you don’t create and delete cluster daily for your use case) |
| 17 | +### Long running clusters (when you don’t create and delete cluster daily for your use case) |
18 | 18 |
|
19 |
| -**Case 1:** When HDInsight clusters are already created without custom virtual network. (No virtual network provided during cluster creation.) |
| 19 | + * **Scenario 1:** Existing HDInsight clusters without custom virtual network. (No virtual network provided during cluster creation.) |
20 | 20 |
|
21 |
| -No immediate disruption until Mar 31, 2025. However, we strongly recommend re-creating the cluster before Mar 31, 2025 to avoid potential disruption. |
| 21 | + * No immediate disruption until Mar 31, 2025. However, we strongly recommend re-creating the cluster before Mar 31, 2025 to avoid potential disruption. |
22 | 22 |
|
23 |
| -**Case 2:** When HDInsight clusters are already created with custom virtual network, no immediate disruption until Mar 31, 2025. However, we strongly recommend re-creating the cluster by Mar 31, 2025 to avoid potential disruption. |
| 23 | + * **Scenario 2:** Existing HDInsight clusters with custom virtual network. |
24 | 24 |
|
25 |
| -### Scenario 2: New clusters creation |
| 25 | + * No immediate disruption until Mar 31, 2025. However, we strongly recommend re-creating the cluster before Mar 31, 2025 to avoid potential disruption. |
26 | 26 |
|
27 |
| -**Case 1:** When HDInsight clusters already created without custom virtual network. (No virtual network provided during cluster creation.) |
| 27 | +### New cluster creation |
28 | 28 |
|
29 |
| -In this case, no impact. You can recreate the cluster directly. |
| 29 | +* **Scenario 1:** HDInsight clusters without custom virtual network. (No virtual network provided during cluster creation) |
30 | 30 |
|
31 |
| -**Case 2:** When HDInsight clusters are created with custom virtual network |
| 31 | + * In this case, no impact. You can recreate the cluster directly. |
32 | 32 |
|
33 |
| -In this case, there are two options to create a cluster |
34 |
| - |
35 |
| -**Approach 1:** Create the cluster with a new subnet |
| 33 | +* **Scenario 2:** HDInsight clusters with custom virtual network. (Providing your own VNet during cluster creation) |
| 34 | + |
| 35 | + * In this case, there are two options to create a cluster |
36 | 36 |
|
37 |
| -1. Choose the outbound connectivity for your cluster, and follow this document [Use Source Network Address Translation (SNAT) for outbound connections](/azure/load-balancer/load-balancer-outbound-connections), and choose one method to provide outbound connectivity for the new cluster. The most recommended way is to attach a NAT gateway and a Network Security Group (NSG) to the subnet. |
| 37 | + **Approach 1:** Create the cluster with a new subnet |
38 | 38 |
|
39 |
| -1. (Optional) Create a new NAT gateway and a new network security group (NSG). |
| 39 | + 1. Choose the outbound connectivity for your cluster, and follow this document [Use Source Network Address Translation (SNAT) for outbound connections](/azure/load-balancer/load-balancer-outbound-connections), and choose one method to provide outbound connectivity for the new cluster. The most recommended way is to attach a NAT gateway and a Network Security Group (NSG) to the subnet. |
| 40 | + |
| 41 | + 1. (Optional) Create a new NAT gateway and a new network security group (NSG). |
40 | 42 |
|
41 |
| - > [!NOTE] |
42 |
| - > You can associate an existing NAT gateway and NSG or the newly created ones. |
43 |
| - > |
44 |
| - > Follow the docs [Quickstart: Create a NAT gateway - Azure portal](/azure/nat-gateway/quickstart-create-nat-gateway-portal), and [Create, change, or delete an Azure network security group](/azure/virtual-network/manage-network-security-group?tabs=network-security-group-portal#create-a-network-security-group) to create a new NAT gateway and a new network security group. |
45 |
| - > |
46 |
| - > You can refer to this document [Control network traffic in Azure HDInsight](./control-network-traffic.md#hdinsight-with-network-security-groups) to set up correct NSG rules. |
| 43 | + > [!NOTE] |
| 44 | + > You can associate an existing NAT gateway and NSG or the newly created ones. |
| 45 | + > |
| 46 | + > Follow the docs [Quickstart: Create a NAT gateway - Azure portal](/azure/nat-gateway/quickstart-create-nat-gateway-portal), and [Create, change, or delete an Azure network security group](/azure/virtual-network/manage-network-security-group?tabs=network-security-group-portal#create-a-network-security-group) to create a new NAT gateway and a new network security group. |
| 47 | + > |
| 48 | + > You can refer to this document [Control network traffic in Azure HDInsight](./control-network-traffic.md#hdinsight-with-network-security-groups) to set up correct NSG rules. |
47 | 49 |
|
48 |
| -1. Create a new subnet and associate the subnet with the NAT gateway and network security group. |
| 50 | + 1. Create a new subnet and associate the subnet with the NAT gateway and network security group. |
49 | 51 |
|
50 |
| - :::image type="content" source="./media/load-balancer-migration-guidelines/create-subnet.png" alt-text="Screenshot showing how to create a subnet." border="true" lightbox="./media/load-balancer-migration-guidelines/create-subnet.png"::: |
51 |
| - |
52 |
| - > [!NOTE] |
53 |
| - > If you are unable to find the NAT gateway, see, [FAQ of the NAT gateway (Azure NAT Gateway frequently asked questions](/azure/nat-gateway/faq#are-basic-sku-resources--basic-load-balancer-and-basic-public-ip-addresses--compatible-with-a-nat-gateway). |
| 52 | + :::image type="content" source="./media/load-balancer-migration-guidelines/create-subnet.png" alt-text="Screenshot showing how to create a subnet." border="true" lightbox="./media/load-balancer-migration-guidelines/create-subnet.png"::: |
| 53 | + |
| 54 | + > [!NOTE] |
| 55 | + > If you are unable to find the NAT gateway, see, [FAQ of the NAT gateway (Azure NAT Gateway frequently asked questions](/azure/nat-gateway/faq#are-basic-sku-resources--basic-load-balancer-and-basic-public-ip-addresses--compatible-with-a-nat-gateway). |
54 | 56 |
|
55 |
| -1. Create a new cluster with the subnet. |
| 57 | + 1. Create a new cluster with the subnet. |
56 | 58 |
|
57 |
| - :::image type="content" source="./media/load-balancer-migration-guidelines/create-hdinsight-cluster.png" alt-text="Screenshot showing how to create a HDInsight cluster." border="true" lightbox="./media/load-balancer-migration-guidelines/create-hdinsight-cluster.png"::: |
| 59 | + :::image type="content" source="./media/load-balancer-migration-guidelines/create-hdinsight-cluster.png" alt-text="Screenshot showing how to create a HDInsight cluster." border="true" lightbox="./media/load-balancer-migration-guidelines/create-hdinsight-cluster.png"::: |
58 | 60 |
|
59 |
| -**Approach 2: Use the existing subnet** |
| 61 | + **Approach 2: Use the existing subnet** |
60 | 62 |
|
61 |
| -To upgrade your existing custom virtual network to integrate Azure standard load balancer (which is HDInsight clusters use by default), see, [Use Source Network Address Translation (SNAT) for outbound connections](/azure/load-balancer/load-balancer-outbound-connections) to provide outbound connectivity for the cluster. The most recommended way is to attach a network security group and a NAT gateway to the subnet. Since the existing subnet which has HDInsight clusters with Azure basic load balancers can't be associated with an NAT gateway due to incompatibility with basic load balancer, there are two scenarios: |
| 63 | + - To upgrade your existing custom virtual network to integrate with Azure standard load balancer (which HDInsight clusters use by default now), see, [Use Source Network Address Translation (SNAT) for outbound connections](/azure/load-balancer/load-balancer-outbound-connections) to provide outbound connectivity for the cluster. |
62 | 64 |
|
63 |
| -### Scenario 1: Existing subnet has no HDInsight clusters with Azure basic load balancer |
| 65 | + The most recommended way is to attach a network security group and a NAT gateway to the subnet. Since the existing subnet which has HDInsight clusters with Azure basic load balancers can't be associated with an NAT gateway due to incompatibility with basic load balancer, there are two scenarios: |
64 | 66 |
|
65 |
| -1. (Optional) Create a new NAT gateway and a new network security group. |
66 |
| - |
67 |
| - > [!NOTE] |
68 |
| - > You could associate an existing NAT gateway and NSG or the newly created ones. |
69 |
| - > Follow this document [Quickstart: Create a NAT gateway - Azure portal](/azure/nat-gateway/quickstart-create-nat-gateway-portal), and [Create, change, or delete an Azure network security group](/azure/virtual-network/manage-network-security-group?tabs=network-security-group-portal#create-a-network-security-group) to create a new NAT gateway and a new network security group. |
70 |
| - > |
71 |
| - > You can refer to this document [Control network traffic in Azure HDInsight](./control-network-traffic.md#hdinsight-with-network-security-groups) to set up correct NSG rules. |
72 |
| - |
73 |
| -1. Associate the NAT gateway with your subnet along with a network security group. |
74 |
| - |
75 |
| - :::image type="content" source="./media/load-balancer-migration-guidelines/associate-gateway.png" alt-text="Screenshot showing how to associate gateway." border="true" lightbox="./media/load-balancer-migration-guidelines/associate-gateway.png"::: |
| 67 | + #### Case 1: Existing subnet has no HDInsight clusters with Azure Basic load balancers |
| 68 | + Follow these steps below: |
76 | 69 |
|
77 |
| -1. Create the cluster with the subnet. |
| 70 | + 1. Create a new NAT gateway and a new network security group or use the existing ones. |
| 71 | + |
| 72 | + > [!NOTE] |
| 73 | + > You could associate an existing NAT gateway and NSG or the newly created ones. |
| 74 | + > Follow this document [Quickstart: Create a NAT gateway - Azure portal](/azure/nat-gateway/quickstart-create-nat-gateway-portal), and [Create, change, or delete an Azure network security group](/azure/virtual-network/manage-network-security-group?tabs=network-security-group-portal#create-a-network-security-group) to create a new NAT gateway and a new network security group. |
| 75 | + > |
| 76 | + > You can refer to this document [Control network traffic in Azure HDInsight](./control-network-traffic.md#hdinsight-with-network-security-groups) to set up correct NSG rules. |
78 | 77 |
|
79 |
| - |
80 |
| - :::image type="content" source="./media/load-balancer-migration-guidelines/security-networking.png" alt-text="Screenshot showing security networking tab." border="true" lightbox="./media/load-balancer-migration-guidelines/security-networking.png"::: |
| 78 | + 1. Associate the NAT gateway with your subnet along with a network security group. |
81 | 79 |
|
82 |
| - > [!NOTE] |
83 |
| - > If you are unable to find the NAT gateway, see, [FAQ of the NAT gateway (Azure NAT Gateway frequently asked questions](/azure/nat-gateway/faq#are-basic-sku-resources--basic-load-balancer-and-basic-public-ip-addresses--compatible-with-a-nat-gateway). |
| 80 | + :::image type="content" source="./media/load-balancer-migration-guidelines/associate-gateway.png" alt-text="Screenshot showing how to associate gateway." border="true" lightbox="./media/load-balancer-migration-guidelines/associate-gateway.png"::: |
84 | 81 |
|
85 |
| -### Scenario 2: The existing subnet has HDInsight clusters with basic load balancers |
| 82 | + 1. Create the cluster with the subnet. |
| 83 | + |
| 84 | + :::image type="content" source="./media/load-balancer-migration-guidelines/security-networking.png" alt-text="Screenshot showing security networking tab." border="true" lightbox="./media/load-balancer-migration-guidelines/security-networking.png"::: |
| 85 | + |
| 86 | + > [!NOTE] |
| 87 | + > If you are unable to find the NAT gateway, see, [FAQ of the NAT gateway (Azure NAT Gateway frequently asked questions](/azure/nat-gateway/faq#are-basic-sku-resources--basic-load-balancer-and-basic-public-ip-addresses--compatible-with-a-nat-gateway). |
86 | 88 |
|
87 |
| -**Approach 1:** The most recommended way is to associate a NAT gateway to the subnet along with network security group. |
| 89 | + #### Case 2: Existing subnet has HDInsight clusters with Azure Basic load balancers |
| 90 | + Consider one of these methods: |
88 | 91 |
|
89 |
| -According to [Azure NAT Gateway frequently asked questions](/azure/nat-gateway/faq#are-basic-sku-resources--basic-load-balancer-and-basic-public-ip-addresses--compatible-with-a-nat-gateway), the subnet with NAT gateway and Azure standard load balancer isn't compatible. |
| 92 | + * **Approach 1:** The most recommended way is to associate a NAT gateway to the subnet along with network security group. |
90 | 93 |
|
91 |
| -To associate with a NAT Gateway, perform the following steps. |
| 94 | + According to [Azure NAT Gateway frequently asked questions](/azure/nat-gateway/faq#are-basic-sku-resources--basic-load-balancer-and-basic-public-ip-addresses--compatible-with-a-nat-gateway), the subnet with NAT gateway and Azure standard load balancer isn't compatible. |
92 | 95 |
|
93 |
| -1. Delete all the existing HDInsight clusters with Azure basic load balancers in this subnet. |
| 96 | + To associate with a NAT Gateway, perform the following steps. |
94 | 97 |
|
95 |
| -1. (Optional) Create a new NAT gateway and a new network security group. |
| 98 | + 1. Delete all the existing HDInsight clusters with Azure basic load balancers in this subnet. |
96 | 99 |
|
97 |
| - > [!NOTE] |
98 |
| - > You could associate an existing NAT gateway and NSG or the newly created ones. |
99 |
| - > Follow this document [Quickstart: Create a NAT gateway - Azure portal](/azure/nat-gateway/quickstart-create-nat-gateway-portal), and [Create, change, or delete an Azure network security group](/azure/virtual-network/manage-network-security-group?tabs=network-security-group-portal#create-a-network-security-group) to create a new NAT gateway and a new network security group. |
100 |
| - > |
101 |
| - > You can refer to this document [Control network traffic in Azure HDInsight](./control-network-traffic.md#hdinsight-with-network-security-groups) to set up correct NSG rules. |
| 100 | + 1. Create a new NAT gateway and a new network security group or use the existing ones. |
102 | 101 |
|
103 |
| -1. Associate the subnet with the NAT gateway and network security group. |
| 102 | + > [!NOTE] |
| 103 | + > You could associate an existing NAT gateway and NSG or use the newly created ones. |
| 104 | + > Follow this document [Quickstart: Create a NAT gateway - Azure portal](/azure/nat-gateway/quickstart-create-nat-gateway-portal), and [Create, change, or delete an Azure network security group](/azure/virtual-network/manage-network-security-group?tabs=network-security-group-portal#create-a-network-security-group) to create a new NAT gateway and a new network security group. |
| 105 | + > |
| 106 | + > You can refer to this document [Control network traffic in Azure HDInsight](./control-network-traffic.md#hdinsight-with-network-security-groups) to set up correct NSG rules. |
104 | 107 |
|
105 |
| - |
106 |
| - :::image type="content" source="./media/load-balancer-migration-guidelines/add-subnet.png" alt-text="Screenshot showing how to add subnet." border="true" lightbox="./media/load-balancer-migration-guidelines/add-subnet.png"::: |
| 108 | + 1. Associate the subnet with the NAT gateway and network security group. |
107 | 109 |
|
108 |
| - > [!NOTE] |
109 |
| - > If you are unable to find the NAT gateway, see, [FAQ of the NAT gateway (Azure NAT Gateway frequently asked questions](/azure/nat-gateway/faq#are-basic-sku-resources--basic-load-balancer-and-basic-public-ip-addresses--compatible-with-a-nat-gateway). |
| 110 | + :::image type="content" source="./media/load-balancer-migration-guidelines/add-subnet.png" alt-text="Screenshot showing how to add subnet." border="true" lightbox="./media/load-balancer-migration-guidelines/add-subnet.png"::: |
| 111 | + |
| 112 | + > [!NOTE] |
| 113 | + > If you are unable to find the NAT gateway, see, [FAQ of the NAT gateway (Azure NAT Gateway frequently asked questions](/azure/nat-gateway/faq#are-basic-sku-resources--basic-load-balancer-and-basic-public-ip-addresses--compatible-with-a-nat-gateway). |
110 | 114 |
|
111 |
| -1. Re-create the clusters with the subnet. |
| 115 | + 1. Re-create the clusters with the subnet. |
112 | 116 |
|
113 |
| - |
114 |
| - :::image type="content" source="./media/load-balancer-migration-guidelines/virtual-network.png" alt-text="Screenshot showing virtual network." border="true" lightbox="./media/load-balancer-migration-guidelines/virtual-network.png"::: |
| 117 | + :::image type="content" source="./media/load-balancer-migration-guidelines/virtual-network.png" alt-text="Screenshot showing virtual network." border="true" lightbox="./media/load-balancer-migration-guidelines/virtual-network.png"::: |
115 | 118 |
|
116 |
| -**Approach 2:** Select option other than **Associate a NAT gateway to the subnet** provided in [Use Source Network Address Translation (SNAT) for outbound connections](/azure/load-balancer/load-balancer-outbound-connections), and follow the instruction for the selected option. |
| 119 | + * **Approach 2:** Select option other than **Associate a NAT gateway to the subnet** provided in [Use Source Network Address Translation (SNAT) for outbound connections](/azure/load-balancer/load-balancer-outbound-connections), and follow the instruction for the selected option. |
117 | 120 |
|
118 |
| -**Approach 3:** Create a new subnet and then create the cluster with the new subnet. |
| 121 | + * **Approach 3:** Create a new subnet and then create the cluster with the new subnet. |
0 commit comments