You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-restrict-outbound-traffic.md
+14-14Lines changed: 14 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,22 +6,22 @@ ms.author: hrasheed
6
6
ms.reviewer: jasonh
7
7
ms.service: hdinsight
8
8
ms.topic: conceptual
9
-
ms.date: 03/11/2020
9
+
ms.date: 04/17/2020
10
10
---
11
11
12
12
# Configure outbound network traffic for Azure HDInsight clusters using Firewall
13
13
14
-
This article provides the steps for you to secure outbound traffic from your HDInsight cluster using Azure Firewall. The steps below assume that you're configuring an Azure Firewall for an existing cluster. If you're deploying a new cluster and behind a firewall, create your HDInsight cluster and subnet first and then follow the steps in this guide.
14
+
This article provides the steps for you to secure outbound traffic from your HDInsight cluster using Azure Firewall. The steps below assume you're configuring an Azure Firewall for an existing cluster. If you're deploying a new cluster behind a firewall, create your HDInsight cluster and subnet first. Then follow the steps in this guide.
15
15
16
16
## Background
17
17
18
-
Azure HDInsight clusters are normally deployed in your own virtual network. The cluster has dependencies on services outside of that virtual network that require network access to function properly.
18
+
HDInsight clusters are normally deployed in a virtual network. The cluster has dependencies on services outside of that virtual network.
19
19
20
20
There are several dependencies that require inbound traffic. The inbound management traffic can't be sent through a firewall device. The source addresses for this traffic are known and are published [here](hdinsight-management-ip-addresses.md). You can also create Network Security Group (NSG) rules with this information to secure inbound traffic to the clusters.
21
21
22
-
The HDInsight outbound traffic dependencies are almost entirely defined with FQDNs, which don't have static IP addresses behind them. The lack of static addresses means that Network Security Groups (NSGs) can't be used to lock down the outbound traffic from a cluster. The addresses change often enough that one can't set up rules based on the current name resolution and use that to set up NSG rules.
22
+
The HDInsight outbound traffic dependencies are almost entirely defined with FQDNs. Which don't have static IP addresses behind them. The lack of static addresses means Network Security Groups (NSGs) can't lock down outbound traffic from a cluster. The addresses change often enough one can't set up rules based on the current name resolution and use.
23
23
24
-
The solution to securing outbound addresses is to use a firewall device that can control outbound traffic based on domain names. Azure Firewall can restrict outbound HTTP and HTTPS traffic based on the FQDN of the destination or [FQDN tags](../firewall/fqdn-tags.md).
24
+
Secure outbound addresses with a firewall that can control outbound traffic based on domain names. Azure Firewall restricts outbound traffic based on the FQDN of the destination or [FQDN tags](../firewall/fqdn-tags.md).
25
25
26
26
## Configuring Azure Firewall with HDInsight
27
27
@@ -69,7 +69,7 @@ Create an application rule collection that allows the cluster to send and receiv
@@ -101,14 +101,14 @@ Create the network rules to correctly configure your HDInsight cluster.
101
101
| --- | --- | --- | --- | --- | --- |
102
102
| Rule_1 | UDP | * | * | 123 | Time service |
103
103
| Rule_2 | Any | * | DC_IP_Address_1, DC_IP_Address_2 | * | If you're using Enterprise Security Package (ESP), then add a network rule in the IP Addresses section that allows communication with AAD-DS for ESP clusters. You can find the IP addresses of the domain controllers on the AAD-DS section in the portal |
104
-
| Rule_3 | TCP | * | IP Address of your Data Lake Storage account | * | If you're using Azure Data Lake Storage, then you can add a network rule in the IP Addresses section to address an SNI issue with ADLS Gen1 and Gen2. This option will route the traffic to firewall, which might result in higher costs for large data loads but the traffic will be logged and auditable in firewall logs. Determine the IP address for your Data Lake Storage account. You can use a powershell command such as `[System.Net.DNS]::GetHostAddresses("STORAGEACCOUNTNAME.blob.core.windows.net")` to resolve the FQDN to an IP address.|
104
+
| Rule_3 | TCP | * | IP Address of your Data Lake Storage account | * | If you're using Azure Data Lake Storage, then you can add a network rule in the IP Addresses section to address an SNI issue with ADLS Gen1 and Gen2. This option will route the traffic to firewall. Which might result in higher costs for large data loads but the traffic will be logged and auditable in firewall logs. Determine the IP address for your Data Lake Storage account. You can use a PowerShell command such as `[System.Net.DNS]::GetHostAddresses("STORAGEACCOUNTNAME.blob.core.windows.net")` to resolve the FQDN to an IP address.|
105
105
| Rule_4 | TCP | * | * | 12000 | (Optional) If you're using Log Analytics, then create a network rule in the IP Addresses section to enable communication with your Log Analytics workspace. |
106
106
107
107
**Service Tags section**
108
108
109
109
| Name | Protocol | Source Addresses | Service Tags | Destination Ports | Notes |
110
110
| --- | --- | --- | --- | --- | --- |
111
-
| Rule_7 | TCP | * | SQL | 1433 | Configure a network rule in the Service Tags section for SQL that will allow you to log and audit SQL traffic, unless you configured Service Endpoints for SQL Server on the HDInsight subnet, which will bypass the firewall. |
111
+
| Rule_7 | TCP | * | SQL | 1433 | Configure a network rule in the Service Tags section for SQL that will allow you to log and audit SQL traffic. Unless you configured Service Endpoints for SQL Server on the HDInsight subnet, which will bypass the firewall. |
112
112
113
113

114
114
@@ -148,7 +148,7 @@ Complete the route table configuration:
148
148
149
149
1. Select **+ Associate**.
150
150
151
-
1. On the **Associate subnet** screen, select the virtual network that your cluster was created into and the **Subnet** you used for your HDInsight cluster.
151
+
1. On the **Associate subnet** screen, select the virtual network that your cluster was created into. And the **Subnet** you used for your HDInsight cluster.
152
152
153
153
1. Select **OK**.
154
154
@@ -166,28 +166,28 @@ If your applications have other dependencies, they need to be added to your Azur
166
166
167
167
Azure Firewall can send logs to a few different storage systems. For instructions on configuring logging for your firewall, follow the steps in [Tutorial: Monitor Azure Firewall logs and metrics](../firewall/tutorial-diagnostics.md).
168
168
169
-
Once you've completed the logging setup, if you're logging data to Log Analytics, you can view blocked traffic with a query such as the following:
169
+
Once you've completed the logging setup, if you're using Log Analytics, you can view blocked traffic with a query such as:
170
170
171
171
```Kusto
172
172
AzureDiagnostics | where msg_s contains "Deny" | where TimeGenerated >= ago(1h)
173
173
```
174
174
175
-
Integrating your Azure Firewall with Azure Monitor logs is useful when first getting an application working when you aren't aware of all of the application dependencies. You can learn more about Azure Monitor logs from [Analyze log data in Azure Monitor](../azure-monitor/log-query/log-query-overview.md)
175
+
Integrating Azure Firewall with Azure Monitor logs is useful when first getting an application working. Especially when you aren't aware of all of the application dependencies. You can learn more about Azure Monitor logs from [Analyze log data in Azure Monitor](../azure-monitor/log-query/log-query-overview.md)
176
176
177
177
To learn about the scale limits of Azure Firewall and request increases, see [this](../azure-resource-manager/management/azure-subscription-service-limits.md#azure-firewall-limits) document or refer to the [FAQs](../firewall/firewall-faq.md).
178
178
179
179
## Access to the cluster
180
180
181
181
After having the firewall set up successfully, you can use the internal endpoint (`https://CLUSTERNAME-int.azurehdinsight.net`) to access the Ambari from inside the virtual network.
182
182
183
-
To use the public endpoint (`https://CLUSTERNAME.azurehdinsight.net`) or ssh endpoint (`CLUSTERNAME-ssh.azurehdinsight.net`), make sure you have the right routes in the route table and NSG rules to avoid the asymmetric routing issue explained [here](../firewall/integrate-lb.md). Specifically in this case, you need to allow the client IP address in the Inbound NSG rules and also add it to the user-defined route table with the next hop set as `internet`. If this isn't set up correctly, you'll see a timeout error.
183
+
To use the public endpoint (`https://CLUSTERNAME.azurehdinsight.net`) or ssh endpoint (`CLUSTERNAME-ssh.azurehdinsight.net`), make sure you have the right routes in the route table and NSG rules to avoid the asymmetric routing issue explained [here](../firewall/integrate-lb.md). Specifically in this case, you need to allow the client IP address in the Inbound NSG rules and also add it to the user-defined route table with the next hop set as `internet`. If the routing isn't set up correctly, you'll see a timeout error.
184
184
185
185
## Configure another network virtual appliance
186
186
187
187
> [!Important]
188
188
> The following information is **only** required if you wish to configure a network virtual appliance (NVA) other than Azure Firewall.
189
189
190
-
The previous instructions help you configure Azure Firewall for restricting outbound traffic from your HDInsight cluster. Azure Firewall is automatically configured to allow traffic for many of the common important scenarios. If you want to use another network virtual appliance, you'll need to manually configure a number of additional features. Keep the following in mind as you configure your network virtual appliance:
190
+
The previous instructions help you configure Azure Firewall for restricting outbound traffic from your HDInsight cluster. Azure Firewall is automatically configured to allow traffic for many of the common important scenarios. Using another network virtual appliance will require you to configure a number of additional features. Keep the following factors in mind as you configure your network virtual appliance:
191
191
192
192
* Service Endpoint capable services should be configured with service endpoints.
193
193
* IP Address dependencies are for non-HTTP/S traffic (both TCP and UDP traffic).
@@ -208,7 +208,7 @@ The previous instructions help you configure Azure Firewall for restricting outb
208
208
|**Endpoint**|**Details**|
209
209
|---|---|
210
210
|\*:123 | NTP clock check. Traffic is checked at multiple endpoints on port 123 |
211
-
| IPs published [here](hdinsight-management-ip-addresses.md)| These are HDInsight service |
211
+
| IPs published [here](hdinsight-management-ip-addresses.md)| These IPs are HDInsight service |
0 commit comments