You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/aks/limit-egress-traffic.md
+65-42Lines changed: 65 additions & 42 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,76 +6,54 @@ author: mlearned
6
6
7
7
ms.service: container-service
8
8
ms.topic: article
9
-
ms.date: 06/06/2019
9
+
ms.date: 08/29/2019
10
10
ms.author: mlearned
11
11
12
12
#Customer intent: As an cluster operator, I want to restrict egress traffic for nodes to only access defined ports and addresses and improve cluster security.
13
13
---
14
14
15
-
# Preview - Limit egress traffic for cluster nodes and control access to required ports and services in Azure Kubernetes Service (AKS)
15
+
# Control egress traffic for cluster nodes in Azure Kubernetes Service (AKS)
16
16
17
-
By default, AKS clusters have unrestricted outbound (egress) internet access. This level of network access allows nodes and services you run to access external resources as needed. If you wish to restrict egress traffic, a limited number of ports and addresses must be accessible to maintain healthy cluster maintenance tasks. Your cluster is then configured to only use base system container images from Microsoft Container Registry (MCR) or Azure Container Registry (ACR), not external public repositories. You must configure your preferred firewall and security rules to allow these required ports and addresses.
17
+
By default, AKS clusters have unrestricted outbound (egress) internet access. This level of network access allows nodes and services you run to access external resources as needed. If you wish to restrict egress traffic, a limited number of ports and addresses must be accessible to maintain healthy cluster maintenance tasks. Your cluster is configured by default to only use base system container images from Microsoft Container Registry (MCR) or Azure Container Registry (ACR). Configure your preferred firewall and security rules to allow these required ports and addresses.
18
18
19
-
This article details what network ports and fully qualified domain names (FQDNs) are required and optional if you restrict egress traffic in an AKS cluster. This feature is currently in preview.
19
+
This article details what network ports and fully qualified domain names (FQDNs) are required and optional if you restrict egress traffic in an AKS cluster.
20
20
21
21
> [!IMPORTANT]
22
-
> AKS preview features are self-service opt-in. Previews are provided "as-is" and "as available" and are excluded from the service level agreements and limited warranty. AKS Previews are partially covered by customer support on best effort basis. As such, these features are not meant for production use. For additional infromation, please see the following support articles:
23
-
>
24
-
> *[AKS Support Policies][aks-support-policies]
25
-
> *[Azure Support FAQ][aks-faq]
22
+
> This document covers only how to lock down the traffic leaving the AKS subnet. AKS has no ingress requirements. Blocking internal subnet traffic using network security groups (NSGs) and firewalls is not supported. To control and block the traffic within the cluster, use [Network Policies][network-policy].
26
23
27
24
## Before you begin
28
25
29
26
You need the Azure CLI version 2.0.66 or later installed and configured. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].
30
27
31
-
To create an AKS cluster that can limit egress traffic, first enable a feature flag on your subscription. This feature registration configures any AKS clusters you create to use base system container images from MCR or ACR. To register the *AKSLockingDownEgressPreview* feature flag, use the [az feature register][az-feature-register] command as shown in the following example:
32
-
33
-
> [!CAUTION]
34
-
> When you register a feature on a subscription, you can't currently un-register that feature. After you enable some preview features, defaults may be used for all AKS clusters then created in the subscription. Don't enable preview features on production subscriptions. Use a separate subscription to test preview features and gather feedback.
35
-
36
-
```azurecli-interactive
37
-
az feature register --name AKSLockingDownEgressPreview --namespace Microsoft.ContainerService
38
-
```
39
-
40
-
It takes a few minutes for the status to show *Registered*. You can check on the registration status by using the [az feature list][az-feature-list] command:
41
-
42
-
```azurecli-interactive
43
-
az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/AKSLockingDownEgressPreview')].{Name:name,State:properties.state}"
44
-
```
45
-
46
-
When ready, refresh the registration of the *Microsoft.ContainerService* resource provider by using the [az provider register][az-provider-register] command:
47
-
48
-
```azurecli-interactive
49
-
az provider register --namespace Microsoft.ContainerService
50
-
```
51
-
52
28
## Egress traffic overview
53
29
54
30
For management and operational purposes, nodes in an AKS cluster need to access certain ports and fully qualified domain names (FQDNs). These actions could be to communicate with the API server, or to download and then install core Kubernetes cluster components and node security updates. By default, egress (outbound) internet traffic is not restricted for nodes in an AKS cluster. The cluster may pull base system container images from external repositories.
55
31
56
-
To increase the security of your AKS cluster, you may wish to restrict egress traffic. The cluster is configured to pull base system container images from MCR or ACR. If you lock down the egress traffic in this manner, you must define specific ports and FQDNs to allow the AKS nodes to correctly communicate with required external services. Without these authorized ports and FQDNs, your AKS nodes can't communicate with the API server or install core components.
32
+
To increase the security of your AKS cluster, you may wish to restrict egress traffic. The cluster is configured to pull base system container images from MCR or ACR. If you lock down the egress traffic in this manner, define specific ports and FQDNs to allow the AKS nodes to correctly communicate with required external services. Without these authorized ports and FQDNs, your AKS nodes can't communicate with the API server or install core components.
57
33
58
34
You can use [Azure Firewall][azure-firewall] or a 3rd-party firewall appliance to secure your egress traffic and define these required ports and addresses. AKS does not automatically create these rules for you. The following ports and addresses are for reference as you create the appropriate rules in your network firewall.
59
35
60
36
> [!IMPORTANT]
61
-
> When you use Azure Firewall to restrict egress traffic and create a user-defined route (UDR) to force all egress traffic, make sure you create an appropriate DNAT rule in Firewall to correctly allow ingress traffic. Using Azure Firewall with a UDR breaks the ingress setup due to asymmetric routing. (The issue occurs because the AKS subnet has a default route that goes to the firewall's private IP address, but you're using a public load balancer - ingress or Kubernetes service of type: LoadBalancer). In this case, the incoming load balancer traffic is received via its public IP address, but the return path goes through the firewall's private IP address. Because the firewall is stateful, it drops the returning packet because the firewall isn't aware of an established session. To learn how to integrate Azure Firewall with your ingress or service load balancer, see [Integrate Azure Firewall with Azure Standard Load Balancer](https://docs.microsoft.com/en-us/azure/firewall/integrate-lb).
62
-
>
37
+
> When you use Azure Firewall to restrict egress traffic and create a user-defined route (UDR) to force all egress traffic, make sure you create an appropriate DNAT rule in Firewall to correctly allow ingress traffic. Using Azure Firewall with a UDR breaks the ingress setup due to asymmetric routing. (The issue occurs if the AKS subnet has a default route that goes to the firewall's private IP address, but you're using a public load balancer - ingress or Kubernetes service of type: LoadBalancer). In this case, the incoming load balancer traffic is received via its public IP address, but the return path goes through the firewall's private IP address. Because the firewall is stateful, it drops the returning packet because the firewall isn't aware of an established session. To learn how to integrate Azure Firewall with your ingress or service load balancer, see [Integrate Azure Firewall with Azure Standard Load Balancer](https://docs.microsoft.com/azure/firewall/integrate-lb).
38
+
>You can lock down the traffic for TCP port 9000 and TCP port 22 using a network rule between the egress worker node IP(s) and the IP for the API server.
63
39
64
40
In AKS, there are two sets of ports and addresses:
65
41
66
42
* The [required ports and address for AKS clusters](#required-ports-and-addresses-for-aks-clusters) details the minimum requirements for authorized egress traffic.
67
43
* The [optional recommended addresses and ports for AKS clusters](#optional-recommended-addresses-and-ports-for-aks-clusters) aren't required for all scenarios, but integration with other services such as Azure Monitor won't work correctly. Review this list of optional ports and FQDNs, and authorize any of the services and components used in your AKS cluster.
68
44
69
45
> [!NOTE]
70
-
> Limiting egress traffic only works on new AKS clusters created after you enable the feature flag registration. For existing clusters, [perform a cluster upgrade operation][aks-upgrade] using the `az aks upgrade` command before you limit the egress traffic.
46
+
> Limiting egress traffic only works on new AKS clusters. For existing clusters, [perform a cluster upgrade operation][aks-upgrade] using the `az aks upgrade` command before you limit the egress traffic.
71
47
72
48
## Required ports and addresses for AKS clusters
73
49
74
50
The following outbound ports / network rules are required for an AKS cluster:
75
51
76
52
* TCP port *443*
53
+
* TCP [IPAddrOfYourAPIServer]:443 is required if you have an app that needs to talk to the API server. This change can be set after the cluster is created.
77
54
* TCP port *9000* and TCP port *22* for the tunnel front pod to communicate with the tunnel end on the API server.
78
55
* To get more specific, see the **.hcp.\<location\>.azmk8s.io* and **.tun.\<location\>.azmk8s.io* addresses in the following table.
56
+
* UDP port *53* for DNS is also required if you have pods directly accessing the API server.
79
57
80
58
The following FQDN / application rules are required:
81
59
@@ -90,23 +68,68 @@ The following FQDN / application rules are required:
90
68
| management.azure.com | HTTPS:443 | This address is required for Kubernetes GET/PUT operations. |
91
69
| login.microsoftonline.com | HTTPS:443 | This address is required for Azure Active Directory authentication. |
92
70
| ntp.ubuntu.com | UDP:123 | This address is required for NTP time synchronization on Linux nodes. |
71
+
| packages.microsoft.com | HTTPS:443 | This address is the Microsoft packages repository used for cached *apt-get* operations. Example packages include Moby, PowerShell, and Azure CLI. |
72
+
| acs-mirror.azureedge.net | HTTPS:443 | This address is for the repository required to install required binaries like kubenet and Azure CNI. |
93
73
94
74
## Optional recommended addresses and ports for AKS clusters
95
75
96
-
* UDP port *53*for DNS
76
+
The following outbound ports / network rules are optional for an AKS cluster:
97
77
98
78
The following FQDN / application rules are recommended for AKS clusters to function correctly:
| security.ubuntu.com, azure.archive.ubuntu.com, changelogs.ubuntu.com | HTTP:80 | This address lets the Linux cluster nodes download the required security patches and updates. |
103
-
| packages.microsoft.com | HTTPS:443 | This address is the Microsoft packages repository used for cached *apt-get* operations. |
104
-
| dc.services.visualstudio.com | HTTPS:443 | Recommended for correct metrics and monitoring using Azure Monitor. |
105
-
|*.opinsights.azure.com | HTTPS:443 | Recommended for correct metrics and monitoring using Azure Monitor. |
106
-
|*.monitoring.azure.com | HTTPS:443 | Recommended for correct metrics and monitoring using Azure Monitor. |
107
-
| gov-prod-policy-data.trafficmanager.net | HTTPS:443 | This address is used for correct operation of Azure Policy (currently in preview in AKS). |
108
-
| apt.dockerproject.org | HTTPS:443 | This address is used for correct driver installation and operation on GPU-based nodes. |
109
-
| nvidia.github.io | HTTPS:443 | This address is used for correct driver installation and operation on GPU-based nodes. |
82
+
| security.ubuntu.com, azure.archive.ubuntu.com, changelogs.ubuntu.com | HTTP:80 | This address lets the Linux cluster nodes download the required security patches and updates. |
83
+
84
+
## Required addresses and ports for GPU enabled AKS clusters
85
+
86
+
The following FQDN / application rules are required for AKS clusters that have GPU enabled:
| dc.services.visualstudio.com | HTTPS:443 | This is for correct metrics and monitoring telemetry using Azure Monitor. |
101
+
|*.ods.opinsights.azure.com | HTTPS:443 | This is used by Azure Monitor for ingesting log analytics data. |
102
+
|*.oms.opinsights.azure.com | HTTPS:443 | This address is used by omsagent, which is used to authenticate the log analytics service. |
103
+
|*.microsoftonline.com | HTTPS:443 | This is used for authenticating and sending metrics to Azure Monitor. |
104
+
|*.monitoring.azure.com | HTTPS:443 | This is used to send metrics data to Azure Monitor. |
105
+
106
+
## Required addresses and ports for AKS clusters with Azure Policy (in public preview) enabled
107
+
108
+
> [!CAUTION]
109
+
> Some of the features below are in preview. The suggestions in this article are subject to change as the feature moves to public preview and future release stages.
110
+
111
+
The following FQDN / application rules are required for AKS clusters that have the Azure Policy enabled.
| gov-prod-policy-data.trafficmanager.net | HTTPS:443 | This address is used for correct operation of Azure Policy. (currently in preview in AKS) |
116
+
| raw.githubusercontent.com | HTTPS:443 | This address is used to pull the built-in policies from GitHub to ensure correct operation of Azure Policy. (currently in preview in AKS) |
117
+
|*.gk.<location>.azmk8s.io | HTTPS:443 | Azure policy add-on talks to Gatekeeper audit endpoint running in master server to get the audit results. |
118
+
| dc.services.visualstudio.com | HTTPS:443 | Azure policy add-on sends telemetry data to applications insights endpoint. |
119
+
120
+
## Required by Windows Server based nodes (in public preview) enabled
121
+
122
+
> [!CAUTION]
123
+
> Some of the features below are in preview. The suggestions in this article are subject to change as the feature moves to public preview and future release stages.
124
+
125
+
The following FQDN / application rules are required for Windows server based AKS clusters:
0 commit comments