Skip to content

Commit 9309d78

Browse files
ShannonLeavittShannonLeavitt
authored
Merge pull request #184126 from nicjohn79/patch-5
Update hdinsight-private-link.md
2 parents ad604f6 + 4e27772 commit 9309d78

File tree

1 file changed

+180
-48
lines changed

1 file changed

+180
-48
lines changed

‎articles/hdinsight/hdinsight-private-link.md

Lines changed: 180 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -19,41 +19,207 @@ The use of Private Link to connect to an HDInsight cluster is an optional featur
1919

2020
When `privateLink` is set to *enabled*, internal [standard load balancers](../load-balancer/load-balancer-overview.md) (SLBs) are created, and an Azure Private Link service is provisioned for each SLB. The Private Link service is what allows you to access the HDInsight cluster from private endpoints.
2121

22-
## Prerequisites
22+
## Private Link Deployment Steps
23+
Successfully creating a Private Link cluster takes many steps, so we have outlined them here. Follow each of the steps below to ensure everything is setup correctly.
2324

24-
Standard load balancers don't automatically provide [public outbound NAT](../load-balancer/load-balancer-outbound-connections.md) as basic load balancers do. You must provide your own NAT solution, such as a NAT gateway or a NAT provided by your [firewall](./hdinsight-restrict-outbound-traffic.md), to connect to outbound, public HDInsight dependencies.
25+
* Step 1: Create prerequisites
26+
* Step 2: Configure HDInsight subnet
27+
* Step 3: Deploy NAT gateway OR firewall
28+
* Step 4: Deploy Private Link cluster
29+
* Step 5: Create private endpoints
30+
* Step 6: Configure DNS
31+
* Step 7: Check cluster connectivity
32+
* Appendix: Manage private endpoints for Azure HDInsight
2533

26-
Your HDInsight cluster still needs access to its outbound dependencies. If these outbound dependencies are not allowed, cluster creation might fail.
34+
## <a name="Createpreqs"></a>Step 1: Create Prerequisites
35+
36+
To start, deploy the following resources if you have not created them already. Once this is done you should have at least 1 resource group, 2 virtual networks, and a network security group to attach to the subnet where the HDInsight cluster will be deployed as shown below.
2737

28-
### Configure a default network security group on the subnet
38+
|Type|Name|Purpose|
39+
|----|----|-------|
40+
|Resource group|hdi-privlink-rg|Used to keep common resources together|
41+
|Virtual network|hdi-privlink-cluster-vnet|The VNET where the cluster will be deployed|
42+
|Virtual network|hdi-privlink-client-vnet|The VNET where clients will connect to the cluster from|
43+
|Network security group|hdi-privlink-cluster-vnet-nsg|Default NSG as required for cluster deployment|
44+
45+
> [!NOTE]
46+
> The network security group (NSG) can simply be deployed, we do not need to modify any NSG rules for cluster deployment.
2947
30-
Create and add a network security group (NSG) on the subnet where you intend to deploy the HDInsight cluster. An NSG is required for enabling outbound connectivity.
3148

32-
### Disable network policies for the Private Link service
49+
## <a name="DisableNetworkPolicy"></a>Step 2: Configure HDInsight Subnet
3350

34-
For the successful creation of a Private Link service, you must explicitly [disable network policies for Private Link services](../private-link/disable-private-link-service-network-policy.md).
51+
In order to choose a source IP address for your Private Link service, an explicit disable setting ```privateLinkServiceNetworkPolicies``` is required on the subnet. Follow the instructions here to [disable network policies for Private Link services](../private-link/disable-private-link-service-network-policy.md).
3552

36-
### Configure a NAT gateway on the subnet
53+
## <a name="NATorFirewall"></a>Step 3: Deploy NAT Gateway *OR* Firewall
3754

38-
You can opt to use a NAT gateway if you don't want to configure a firewall or a network virtual appliance (NVA) for NAT. Otherwise, skip to the next prerequisite.
55+
Standard load balancers don't automatically provide [public outbound NAT](../load-balancer/load-balancer-outbound-connections.md) as basic load balancers do. Since Private Link clusters use standard load balancers, you must provide your own NAT solution, such as a NAT gateway or a NAT provided by your [firewall](./hdinsight-restrict-outbound-traffic.md), to connect to outbound, public HDInsight dependencies.
3956

40-
To get started, add a NAT gateway (with a new public IP address in your virtual network) to the configured subnet of your virtual network. This gateway is responsible for translating your private internal IP address to public addresses when traffic needs to go outside your virtual network.
57+
### Deploy a NAT Gateway (Option 1)
58+
You can opt to use a NAT gateway if you don't want to configure a firewall or a network virtual appliance (NVA) for NAT. To get started, add a NAT gateway (with a new public IP address in your virtual network) to the configured subnet of your virtual network. This gateway is responsible for translating your private internal IP address to public addresses when traffic needs to go outside your virtual network.
4159

42-
### Configure a firewall (optional)
60+
For a basic setup to get started:
61+
62+
1. Search for 'NAT Gateways' in the Azure portal and click **Create**.
63+
2. Use the following configurations in the NAT Gateway. (We are not including all configs here, so you can use the default value for those)
64+
65+
| Config | Value |
66+
| ------ | ----- |
67+
| NAT gateway name | hdi-privlink-nat-gateway |
68+
| Public IP Prefixes | Create a new public IP prefix |
69+
| Public IP prefix name | hdi-privlink-nat-gateway-prefix |
70+
| Public IP prefix size | /28 (16 addresses) |
71+
| Virtual network | hdi-privlink-cluster-vnet |
72+
| Subnet name | default |
73+
74+
3. Once the NAT Gateway is finished deploying, you are ready to go to the next step.
75+
76+
### Configure a firewall (Option 2)
4377
For a basic setup to get started:
4478

4579
1. Add a new subnet named *AzureFirewallSubnet* to your virtual network.
4680
1. Use the new subnet to configure a new firewall and add your firewall policies.
4781
1. Use the new firewall's private IP address as the `nextHopIpAddress` value in your route table.
4882
1. Add the route table to the configured subnet of your virtual network.
4983

84+
Your HDInsight cluster still needs access to its outbound dependencies. If these outbound dependencies are not allowed, cluster creation might fail.
5085
For more information on setting up a firewall, see [Control network traffic in Azure HDInsight](./control-network-traffic.md).
5186

52-
The following diagram shows an example of the networking configuration that's required before you create a cluster. In this example, all outbound traffic is forced to Azure Firewall through a user-defined route. The required outbound dependencies should be allowed on the firewall before cluster creation. For Enterprise Security Package clusters, virtual network peering can provide the network connectivity to Azure Active Directory Domain Services.
87+
## <a name="deployCluster"></a>Step 4: Deploy Private Link cluster
88+
89+
At this point all prerequisites should be taken care of and you are ready to deploy the Private Link cluster. The following diagram shows an example of the networking configuration that's required before you create the cluster. In this example, all outbound traffic is forced to Azure Firewall through a user-defined route. The required outbound dependencies should be allowed on the firewall before cluster creation. For Enterprise Security Package clusters, virtual network peering can provide the network connectivity to Azure Active Directory Domain Services.
5390

5491
:::image type="content" source="media/hdinsight-private-link/before-cluster-creation.png" alt-text="Diagram of the Private Link environment before cluster creation.":::
5592

56-
## Manage private endpoints for Azure HDInsight
93+
### Create the cluster
94+
95+
The following JSON code snippet includes the two network properties that you must configure in your Azure Resource Manager template to create a private HDInsight cluster:
96+
97+
```json
98+
networkProperties: {
99+
"resourceProviderConnection": "Outbound",
100+
"privateLink": "Enabled"
101+
}
102+
```
103+
For a complete template with many of the HDInsight enterprise security features, including Private Link, see [HDInsight enterprise security template](https://github.com/Azure-Samples/hdinsight-enterprise-security/tree/main/ESP-HIB-PL-Template).
104+
105+
To create a cluster by using PowerShell, see the [example](/powershell/module/az.hdinsight/new-azhdinsightcluster#example-4--create-an-azure-hdinsight-cluster-with-relay-outbound-and-private-link-feature).
106+
107+
To create a cluster by using the Azure CLI, see the [example](/cli/azure/hdinsight#az_hdinsight_create-examples).
108+
109+
## <a name="PrivateEndpoints"></a>Step 5: Create Private Endpoints
110+
111+
Azure automatically creates a Private link service for the Ambari and SSH load balancers during the Private Link cluster deployment. After the cluster is deployed, you have to create two Private endpoints on the client VNET(s), one for Ambari and one for SSH access. Then, link them to the Private link services which were created as part of the cluster deployment.
112+
113+
To create the Private Endpoints:
114+
1. Open the Azure portal and search for 'Private link'.
115+
2. In the results, click the Private link icon.
116+
3. Click 'Create private endpoint' and use the following configurations to setup the Ambari private endpoint:
117+
118+
| Config | Value |
119+
| ------ | ----- |
120+
| Name | hdi-privlink-cluster |
121+
| Resource type | Microsoft.Network/privateLinkServices |
122+
| Resource | gateway-* (This should match the HDI deployment ID of your cluster, for example gateway-4eafe3a2a67e4cd88762c22a55fe4654) |
123+
| Virtual network | hdi-privlink-client-vnet |
124+
| Subnet | default |
125+
126+
4. Repeat the process to create another private endpoint for SSH access using the following configurations:
127+
128+
| Config | Value |
129+
| ------ | ----- |
130+
| Name | hdi-privlink-cluster-ssh |
131+
| Resource type | Microsoft.Network/privateLinkServices |
132+
| Resource | headnode-* (This should match the HDI deployment ID of your cluster, for example headnode-4eafe3a2a67e4cd88762c22a55fe4654) |
133+
| Virtual network | hdi-privlink-client-vnet |
134+
| Subnet | default |
135+
136+
Once the private endpoints are created, you’re done with this phase of the setup. If you didn’t make a note of the private IP addresses assigned to the endpoints, follow the steps below:
137+
138+
1. Open the client VNET in the Azure portal.
139+
2. Click the 'Overview' tab.
140+
3. You should see both the Ambari and ssh Network interfaces listed and their private IP Addresses.
141+
4. Make a note of these IP addresses because they are required to connect to the cluster and properly configure DNS.
142+
143+
## <a name="ConfigureDNS"></a>Step 6: Configure DNS to connect over private endpoints
144+
145+
To access private clusters, you can configure DNS resolution through private DNS zones. The Private Link entries created in the Azure-managed public DNS zone `azurehdinsight.net` are as follows:
146+
147+
```dns
148+
<clustername> CNAME <clustername>.privatelink
149+
<clustername>-int CNAME <clustername>-int.privatelink
150+
<clustername>-ssh CNAME <clustername>-ssh.privatelink
151+
```
152+
The following image shows an example of the private DNS entries configured to enable access to a cluster from a virtual network that isn't peered or doesn't have a direct line of sight to the cluster. You can use an Azure DNS private zone to override `*.privatelink.azurehdinsight.net` fully qualified domain names (FQDNs) and resolve private endpoints' IP addresses in the client's network. The configuration is only for `<clustername>.azurehdinsight.net` in the example, but it also extends to other cluster endpoints.
153+
154+
:::image type="content" source="media/hdinsight-private-link/access-private-clusters.png" alt-text="Diagram of the Private Link architecture.":::
155+
156+
To configure DNS resolution through a Private DNS zone:
157+
158+
1. Create an Azure Private DNS zone. (We are not including all configs here, all other configs are left at default values)
159+
160+
| Config | Value |
161+
| ------ | ----- |
162+
| Name | privatelink.azurehdinsight.net |
163+
164+
2. Add a Record set to the Private DNS zone for Ambari.
165+
166+
| Config | Value |
167+
| ------ | ----- |
168+
| Name | YourPrivateLinkClusterName |
169+
| Type | A - Alias record to IPv4 address |
170+
| TTL | 1 |
171+
| TTL unit | Hours |
172+
| IP Address | Private IP of private endpoint for Ambari access |
173+
174+
3. Add a Record set to the Private DNS zone for SSH.
175+
176+
| Config | Value |
177+
| ------ | ----- |
178+
| Name | YourPrivateLinkClusterName-ssh |
179+
| Type | A - Alias record to IPv4 address |
180+
| TTL | 1 |
181+
| TTL unit | Hours |
182+
| IP Address | Private IP of private endpoint for SSH access |
183+
184+
4. Associate the private DNS zone with the client VNET by adding a Virtual Network Link.
185+
1. Open the private DNS zone in the Azure portal.
186+
1. Click the 'Virtual network links' tab.
187+
1. Click the 'Add' button.
188+
1. Fill in the details: Link name, Subscription, and Virtual Network
189+
1. Click **Save**.
190+
191+
## <a name="CheckConnectivity"></a>Step 6: Check cluster connectivity
192+
193+
The last step is to test connectivity to the cluster. Since this cluster is isolated or private, we cannot access the cluster using any public IP or FQDN. Instead we have a couple of options:
194+
195+
* Set up VPN access to the client VNET from your on premise network
196+
* Deploy a VM to the client VNET and access the cluster from this VM
197+
198+
For this example, we will deploy a VM in the client VNET using the following configuration to test the connectivity.
199+
200+
| Config | Value |
201+
| ------ | ----- |
202+
| Virtual machine name | hdi-privlink-client-vm |
203+
| Image | Windows 10 Pro, Version 2004 - Gen1 |
204+
| Public inbound ports | Allow selected ports |
205+
| Select inbound ports | RDP (3389) |
206+
| I confirm I have an eligible Windows 10 license... | Checked |
207+
| Virtual network | hdi-privlink-client-vnet |
208+
| Subnet | default |
209+
210+
Once the client VM is deployed, you can test both Ambari and SSH access.
211+
212+
To test Ambari access: <br>
213+
1. Open a web browser on the VM.
214+
2. Navigate to your cluster's regular FQDN: `https://<clustername>.azurehdinsight.net`
215+
3. If the Ambari UI loads, the configuration is correct for Ambari access.
216+
217+
To test ssh access: <br>
218+
1. Open a command prompt to get a terminal window.
219+
2. In the terminal window, try connecting to your cluster with SSH: `ssh sshuser@<clustername>.azurehdinsight.net` (Replace "sshuser" with the ssh user you created for your cluster)
220+
3. If you are able to connect, the configuration is correct for SSH access.
221+
222+
## <a name="ManageEndpoints"></a>Manage Private endpoints for Azure HDInsight
57223

58224
You can use [private endpoints](../private-link/private-endpoint-overview.md) for your Azure HDInsight clusters to allow clients on a virtual network to securely access your cluster over [Private Link](../private-link/private-link-overview.md). Network traffic between the clients on the virtual network and the HDInsight cluster traverses over the Microsoft backbone network, eliminating exposure from the public internet.
59225

@@ -77,41 +243,7 @@ The following table shows the various HDInsight resource actions and the resulti
77243
| Reject | Rejected | Connection was rejected by the Private Link resource owner. |
78244
| Remove | Disconnected | Connection was removed by the Private Link resource owner. The private endpoint becomes informative and should be deleted for cleanup. |
79245

80-
## Configure DNS to connect over private endpoints
81-
82-
After you've set up the networking, you can create a cluster with an outbound resource provider connection and Private Link enabled.
83-
84-
To access private clusters, you can use Private Link DNS extensions and private endpoints. When `privateLink` is set to *enabled*, you can create private endpoints and configure DNS resolution through private DNS zones.
85-
86-
The Private Link entries created in the Azure-managed public DNS zone `azurehdinsight.net` are as follows:
87-
88-
```dns
89-
<clustername> CNAME <clustername>.privatelink
90-
<clustername>-int CNAME <clustername>-int.privatelink
91-
<clustername>-ssh CNAME <clustername>-ssh.privatelink
92-
```
93-
The following image shows an example of the private DNS entries configured to enable access to a cluster from a virtual network that isn't peered or doesn't have a direct line of sight to the cluster. You can use an Azure DNS private zone to override `*.privatelink.azurehdinsight.net` fully qualified domain names (FQDNs) and resolve private endpoints' IP addresses in the client's network. The configuration is only for `<clustername>.azurehdinsight.net` in the example, but it also extends to other cluster endpoints.
94-
95-
:::image type="content" source="media/hdinsight-private-link/access-private-clusters.png" alt-text="Diagram of the Private Link architecture.":::
96-
97-
## Create clusters
98-
99-
The following JSON code snippet includes the two network properties that you must configure in your Azure Resource Manager template to create a private HDInsight cluster:
100-
101-
```json
102-
networkProperties: {
103-
"resourceProviderConnection": "Outbound",
104-
"privateLink": "Enabled"
105-
}
106-
```
107-
108-
For a complete template with many of the HDInsight enterprise security features, including Private Link, see [HDInsight enterprise security template](https://github.com/Azure-Samples/hdinsight-enterprise-security/tree/main/ESP-HIB-PL-Template).
109-
110-
To create a cluster by using PowerShell, see the [example](/powershell/module/az.hdinsight/new-azhdinsightcluster#example-4--create-an-azure-hdinsight-cluster-with-relay-outbound-and-private-link-feature).
111-
112-
To create a cluster by using the Azure CLI, see the [example](/cli/azure/hdinsight#az_hdinsight_create-examples).
113-
114246
## Next steps
115247

116248
* [Enterprise Security Package for Azure HDInsight](enterprise-security-package.md)
117-
* [Enterprise security general information and guidelines in Azure HDInsight](./domain-joined/general-guidelines.md)
249+
* [Enterprise security general information and guidelines in Azure HDInsight](./domain-joined/general-guidelines.md)

0 commit comments

Comments
 (0)