Merge pull request #106978 from palma21/egress-changes

[AKS] New Egress Requirements

Author: PRMerger9

articles/aks/limit-egress-traffic.md

@@ -32,7 +32,7 @@ You can use [Azure Firewall][azure-firewall] or a 3rd-party firewall appliance t
 
 > [!IMPORTANT]
 > When you use Azure Firewall to restrict egress traffic and create a user-defined route (UDR) to force all egress traffic, make sure you create an appropriate DNAT rule in Firewall to correctly allow ingress traffic. Using Azure Firewall with a UDR breaks the ingress setup due to asymmetric routing. (The issue occurs if the AKS subnet has a default route that goes to the firewall's private IP address, but you're using a public load balancer - ingress or Kubernetes service of type: LoadBalancer). In this case, the incoming load balancer traffic is received via its public IP address, but the return path goes through the firewall's private IP address. Because the firewall is stateful, it drops the returning packet because the firewall isn't aware of an established session. To learn how to integrate Azure Firewall with your ingress or service load balancer, see [Integrate Azure Firewall with Azure Standard Load Balancer](https://docs.microsoft.com/azure/firewall/integrate-lb).
-> You can lock down the traffic for TCP port 9000 and TCP port 22 using a network rule between the egress worker node IP(s) and the IP for the API server.
+> You can lock down the traffic for TCP port 9000, TCP port 22 and UDP port 1194 using a network rule between the egress worker node IP(s) and the IP for the API server.
 
 In AKS, there are two sets of ports and addresses:
 
@@ -48,7 +48,7 @@ The following outbound ports / network rules are required for an AKS cluster:
 
 * TCP port *443*
 * TCP [IPAddrOfYourAPIServer]:443 is required if you have an app that needs to talk to the API server.  This change can be set after the cluster is created.
-* TCP port *9000* and TCP port *22* for the tunnel front pod to communicate with the tunnel end on the API server.
+* TCP port *9000*, TCP port *22* and UDP port *1194* for the tunnel front pod to communicate with the tunnel end on the API server.
     * To get more specific, see the **.hcp.\<location\>.azmk8s.io* and **.tun.\<location\>.azmk8s.io* addresses in the following table.
 * UDP port *123* for Network Time Protocol (NTP) time synchronization (Linux nodes).
 * UDP port *53* for DNS is also required if you have pods directly accessing the API server.
@@ -59,31 +59,32 @@ The following FQDN / application rules are required:
 > ***.blob.core.windows.net and aksrepos.azurecr.io** are no longer required FQDN rules for egress lockdown.  For existing clusters, [perform a cluster upgrade operation][aks-upgrade] using the `az aks upgrade` command to remove these rules.
 
 > [!IMPORTANT]
-> *.cdn.mscr.io has been replaced by a *.data.mcr.microsoft.com for the Azure public cloud regions. Please upgdate your existing firewall rules for the changes to take effect.
+> *.cdn.mscr.io has been replaced by *.data.mcr.microsoft.com for the Azure public cloud regions. Please upgrade your existing firewall rules for the changes to take effect.
 
 - Azure Global
 
 | FQDN                       | Port      | Use      |
 |----------------------------|-----------|----------|
-| *.hcp.\<location\>.azmk8s.io | HTTPS:443, TCP:22, TCP:9000 | This address is the API server endpoint. Replace *\<location\>* with the region where your AKS cluster is deployed. |
-| *.tun.\<location\>.azmk8s.io | HTTPS:443, TCP:22, TCP:9000 | This address is the API server endpoint. Replace *\<location\>* with the region where your AKS cluster is deployed. |
+| *.hcp.\<location\>.azmk8s.io | HTTPS:443, TCP:22, TCP:9000, UDP:1194 | This address is required for Node <-> API server communication. Replace *\<location\>* with the region where your AKS cluster is deployed. |
+| *.tun.\<location\>.azmk8s.io | HTTPS:443, TCP:22, TCP:9000, UDP:1194 | This address is required for Node <-> API server communication. Replace *\<location\>* with the region where your AKS cluster is deployed. |
 | mcr.microsoft.com          | HTTPS:443 | This address is required to access images in Microsoft Container Registry (MCR). This registry contains first-party images/charts(for example, moby, etc.) required for the functioning of the cluster during upgrade and scale of the cluster |
 | *.data.mcr.microsoft.com             | HTTPS:443 | This address is required for MCR storage backed by the Azure content delivery network (CDN). |
 | management.azure.com       | HTTPS:443 | This address is required for Kubernetes GET/PUT operations. |
 | login.microsoftonline.com  | HTTPS:443 | This address is required for Azure Active Directory authentication. |
 | ntp.ubuntu.com             | UDP:123   | This address is required for NTP time synchronization on Linux nodes. |
 | packages.microsoft.com     | HTTPS:443 | This address is the Microsoft packages repository used for cached *apt-get* operations.  Example packages include Moby, PowerShell, and Azure CLI. |
-| acs-mirror.azureedge.net 	 | HTTPS:443 | This address is for the repository required to install required binaries like kubenet and Azure CNI. |
+| acs-mirror.azureedge.net      | HTTPS:443 | This address is for the repository required to install required binaries like kubenet and Azure CNI. |
 
 - Azure China 21Vianet
 
 | FQDN                       | Port      | Use      |
 |----------------------------|-----------|----------|
-| *.hcp.\<location\>.cx.prod.service.azk8s.cn | HTTPS:443, TCP:22, TCP:9000 | This address is the API server endpoint. Replace *\<location\>* with the region where your AKS cluster is deployed. |
-| *.tun.\<location\>.cx.prod.service.azk8s.cn | HTTPS:443, TCP:22, TCP:9000 | This address is the API server endpoint. Replace *\<location\>* with the region where your AKS cluster is deployed. |
+| *.hcp.\<location\>.cx.prod.service.azk8s.cn | HTTPS:443, TCP:22, TCP:9000, UDP:1194 | This address is required for Node <-> API server communication. Replace *\<location\>* with the region where your AKS cluster is deployed. |
+| *.tun.\<location\>.cx.prod.service.azk8s.cn | HTTPS:443, TCP:22, TCP:9000, UDP:1194 | This address is required for Node <-> API server communication. Replace *\<location\>* with the region where your AKS cluster is deployed. |
 | *.azk8s.cn        | HTTPS:443 | This address is required to download required binaries and images|
 | mcr.microsoft.com          | HTTPS:443 | This address is required to access images in Microsoft Container Registry (MCR). This registry contains first-party images/charts(for example, moby, etc.) required for the functioning of the cluster during upgrade and scale of the cluster |
 | *.cdn.mscr.io       | HTTPS:443 | This address is required for MCR storage backed by the Azure Content Delivery Network (CDN). |
+| *.data.mcr.microsoft.com             | HTTPS:443 | This address is required for MCR storage backed by the Azure content delivery network (CDN). |
 | management.chinacloudapi.cn       | HTTPS:443 | This address is required for Kubernetes GET/PUT operations. |
 | login.chinacloudapi.cn  | HTTPS:443 | This address is required for Azure Active Directory authentication. |
 | ntp.ubuntu.com             | UDP:123   | This address is required for NTP time synchronization on Linux nodes. |
@@ -93,15 +94,16 @@ The following FQDN / application rules are required:
 
 | FQDN                       | Port      | Use      |
 |----------------------------|-----------|----------|
-| *.hcp.\<location\>.cx.aks.containerservice.azure.us | HTTPS:443, TCP:22, TCP:9000 | This address is the API server endpoint. Replace *\<location\>* with the region where your AKS cluster is deployed. |
-| *.tun.\<location\>.cx.aks.containerservice.azure.us | HTTPS:443, TCP:22, TCP:9000 | This address is the API server endpoint. Replace *\<location\>* with the region where your AKS cluster is deployed. |
+| *.hcp.\<location\>.cx.aks.containerservice.azure.us | HTTPS:443, TCP:22, TCP:9000, UDP:1194 | This address is required for Node <-> API server communication. Replace *\<location\>* with the region where your AKS cluster is deployed. |
+| *.tun.\<location\>.cx.aks.containerservice.azure.us | HTTPS:443, TCP:22, TCP:9000, UDP:1194 | This address is required for Node <-> API server communication. Replace *\<location\>* with the region where your AKS cluster is deployed. |
 | mcr.microsoft.com          | HTTPS:443 | This address is required to access images in Microsoft Container Registry (MCR). This registry contains first-party images/charts(for example, moby, etc.) required for the functioning of the cluster during upgrade and scale of the cluster |
 |*.cdn.mscr.io              | HTTPS:443 | This address is required for MCR storage backed by the Azure Content Delivery Network (CDN). |
+| *.data.mcr.microsoft.com             | HTTPS:443 | This address is required for MCR storage backed by the Azure content delivery network (CDN). |
 | management.usgovcloudapi.net       | HTTPS:443 | This address is required for Kubernetes GET/PUT operations. |
 | login.microsoftonline.us  | HTTPS:443 | This address is required for Azure Active Directory authentication. |
 | ntp.ubuntu.com             | UDP:123   | This address is required for NTP time synchronization on Linux nodes. |
 | packages.microsoft.com     | HTTPS:443 | This address is the Microsoft packages repository used for cached *apt-get* operations.  Example packages include Moby, PowerShell, and Azure CLI. |
-| acs-mirror.azureedge.net 	 | HTTPS:443 | This address is for the repository required to install required binaries like kubenet and Azure CNI. |
+| acs-mirror.azureedge.net      | HTTPS:443 | This address is for the repository required to install required binaries like kubenet and Azure CNI. |
 
 ## Optional recommended addresses and ports for AKS clusters
 
@@ -129,8 +131,8 @@ The following FQDN / application rules are required for AKS clusters that have t
 
 | FQDN                                    | Port      | Use      |
 |-----------------------------------------|-----------|----------|
-| dc.services.visualstudio.com | HTTPS:443	| This is for correct metrics and monitoring telemetry using Azure Monitor. |
-| *.ods.opinsights.azure.com	| HTTPS:443	| This is used by Azure Monitor for ingesting log analytics data. |
+| dc.services.visualstudio.com | HTTPS:443    | This is for correct metrics and monitoring telemetry using Azure Monitor. |
+| *.ods.opinsights.azure.com    | HTTPS:443    | This is used by Azure Monitor for ingesting log analytics data. |
 | *.oms.opinsights.azure.com | HTTPS:443 | This address is used by omsagent, which is used to authenticate the log analytics service. |
 |*.microsoftonline.com | HTTPS:443 | This is used for authenticating and sending metrics to Azure Monitor. |
 |*.monitoring.azure.com | HTTPS:443 | This is used to send metrics data to Azure Monitor. |
@@ -157,7 +159,7 @@ The following FQDN / application rules are required for AKS clusters that have t
 |-----------------------------------------|-----------|----------|
 | gov-prod-policy-data.trafficmanager.net | HTTPS:443 | This address is used for correct operation of Azure Policy. (currently in preview in AKS) |
 | raw.githubusercontent.com | HTTPS:443 | This address is used to pull the built-in policies from GitHub to ensure correct operation of Azure Policy. (currently in preview in AKS) |
-| *.gk.\<location\>.azmk8s.io | HTTPS:443	| Azure policy add-on that talks to Gatekeeper audit endpoint running in master server to get the audit results. |
+| *.gk.\<location\>.azmk8s.io | HTTPS:443    | Azure policy add-on that talks to Gatekeeper audit endpoint running in master server to get the audit results. |
 | dc.services.visualstudio.com | HTTPS:443 | Azure policy add-on that sends telemetry data to applications insights endpoint. |
 
 ## Required by Windows Server based nodes (in public preview) enabled