Merge pull request #102389 from zr-msft/aks-alloc-port

[AKS] add AllocatedOutboundPorts and IdleTimeoutInMinutes callout

Author: v-albemi

articles/aks/load-balancer-standard.md

@@ -24,11 +24,11 @@ If you don't have an Azure subscription, create a [free account](https://azure.m
 
 [!INCLUDE [cloud-shell-try-it.md](../../includes/cloud-shell-try-it.md)]
 
-If you choose to install and use the CLI locally, this article requires that you are running the Azure CLI version 2.0.74 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].
+If you choose to install and use the CLI locally, this article requires that you are running the Azure CLI version 2.0.81 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].
 
 ## Before you begin
 
-This article assumes you have a AKS cluster with the *Standard* SKU Azure Load Balancer. If you need an AKS cluster, see the AKS quickstart [using the Azure CLI][aks-quickstart-cli] or [using the Azure portal][aks-quickstart-portal].
+This article assumes you have an AKS cluster with the *Standard* SKU Azure Load Balancer. If you need an AKS cluster, see the AKS quickstart [using the Azure CLI][aks-quickstart-cli] or [using the Azure portal][aks-quickstart-portal].
 
 The AKS cluster service principal needs also permission to manage network resources if you use an existing subnet or resource group. In general, assign the *Network contributor* role to your service principal on the delegated resources. For more information on permissions, see [Delegate AKS access to other Azure resources][aks-sp].
 
@@ -65,7 +65,7 @@ You can also configure the load balancer to be internal and not expose a public
 
 When using a *Standard* SKU load balancer with managed outbound public IPs, which are created by default, you can scale the number of managed outbound public IPs using the *load-balancer-managed-ip-count* parameter.
 
-To update an existing cluster run the following command. This parameter can also be set at cluster create-time to have multiple managed outbound public IPs.
+To update an existing cluster, run the following command. This parameter can also be set at cluster create-time to have multiple managed outbound public IPs.
 
 ```azurecli-interactive
 az aks update \
@@ -160,9 +160,14 @@ az aks create \
     --load-balancer-outbound-ip-prefixes <publicIpPrefixId1>,<publicIpPrefixId2>
 ```
 
-## Show the outbound rule for your load balancer
+## Configure outbound allocated ports and idle timeout
 
-To show the outbound rule created in the load balancer, use [az network lb outbound-rule list][az-network-lb-outbound-rule-list] and specify the node resource group of your AKS cluster:
+> [!WARNING]
+> The following section is intended for advanced scenarios of larger scale networking or for addressing SNAT exhaustion issues with the default configurations. You must have an accurate inventory of available quota for VMs and IP addresses before changing *AllocatedOutboundPorts* or *IdleTimeoutInMinutes* from their default value in order to maintain healthy clusters.
+> 
+> Altering the values for *AllocatedOutboundPorts* and *IdleTimeoutInMinutes* may significantly change the behavior of the outbound rule for your load balancer. Review the [Load Balancer outbound rules][azure-lb-outbound-rules-overview], [load Balancer outbound rules][azure-lb-outbound-rules], and [outbound connections in Azure][azure-lb-outbound-connections] before updating these values to fully understand the impact of your changes.
+
+Outbound allocated ports and their idle timeouts are used for [SNAT][azure-lb-outbound-connections]. By default, the *Standard* SKU load balancer uses [automatic assignment for the number of outbound ports based on backend pool size][azure-lb-outbound-preallocatedports] and a 30-minute idle timeout for each port. To see these values, use [az network lb outbound-rule list][az-network-lb-outbound-rule-list] to show the outbound rule for the load balancer:
 
 ```azurecli-interactive
 NODE_RG=$(az aks show --resource-group myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv)
@@ -177,7 +182,46 @@ AllocatedOutboundPorts    EnableTcpReset    IdleTimeoutInMinutes    Name
 0                         True              30                      aksOutboundRule  All         Succeeded            MC_myResourceGroup_myAKSCluster_eastus  
 ```
 
-In the example output, *AllocatedOutboundPorts* is 0. The value for *AllocatedOutboundPorts* means that SNAT port allocation reverts to automatic assignment based on backend pool size. See [Load Balancer outbound rules][azure-lb-outbound-rules] and [Outbound connections in Azure][azure-lb-outbound-connections] for more details.
+The example output shows the default value for *AllocatedOutboundPorts* and *IdleTimeoutInMinutes*. A value of 0 for *AllocatedOutboundPorts* sets the number of outbound ports using automatic assignment for the number of outbound ports based on backend pool size. For example, if the cluster has 50 or less nodes, 1024 ports for each node are allocated.
+
+Consider changing the setting of *allocatedOutboundPorts* or *IdleTimeoutInMinutes* if you expect to face SNAT exhaustion based on the above default configuration. Each additional IP address enables 64,000 additional ports for allocation, however the Azure Standard Load Balancer does not automatically increase the ports per node when more IP addresses are added. You can change these values by setting the *load-balancer-outbound-ports* and *load-balancer-idle-timeout* parameters. For example:
+
+```azurecli-interactive
+az aks update \
+    --resource-group myResourceGroup \
+    --name myAKSCluster \
+    --load-balancer-outbound-ports 0 \
+    --load-balancer-idle-timeout 30
+```
+
+> [!IMPORTANT]
+> You must [calculate your required quota][calculate-required-quota] before customizing *allocatedOutboundPorts* to avoid connectivity or scaling issues. The value you specify for *allocatedOutboundPorts* must also be a multiple of 8.
+
+You can also use the *load-balancer-outbound-ports* and *load-balancer-idle-timeout* parameters when creating a cluster, but you must also specify either *load-balancer-managed-outbound-ip-count*, *load-balancer-outbound-ips*, or *load-balancer-outbound-ip-prefixes* as well.  For example:
+
+```azurecli-interactive
+az aks create \
+    --resource-group myResourceGroup \
+    --name myAKSCluster \
+    --vm-set-type VirtualMachineScaleSets \
+    --node-count 1 \
+    --load-balancer-sku standard \
+    --generate-ssh-keys \
+    --load-balancer-managed-outbound-ip-count 2 \
+    --load-balancer-outbound-ports 0 \
+    --load-balancer-idle-timeout 30
+```
+
+When altering the *load-balancer-outbound-ports* and *load-balancer-idle-timeout* parameters from their default, it affects the behavior of the load balancer profile, which impacts the entire cluster.
+
+### Required quota for customizing allocatedOutboundPorts
+You must have enough outbound IP capacity based on the number of your node VMs and desired allocated outbound ports. To validate you have enough outbound IP capacity, use the following formula: 
+ 
+*outboundIPs* \* 64,000 \> *nodeVMs* \* *desiredAllocatedOutboundPorts*.
+ 
+For example, if you have 3 *nodeVMs*, and 50,000 *desiredAllocatedOutboundPorts*, you need to have at least 3 *outboundIPs*. It is recommended that you incorporate additional outbound IP capacity beyond what you need. Additionally, you must account for the cluster autoscaler and the possibility of node pool upgrades when calculating outbound IP capacity. For the cluster autoscaler, review the current node count and the maximum node count and use the higher value. For upgrading, account for an additional node VM for every node pool that allows upgrading.
+ 
+When setting *IdleTimeoutInMinutes* to a different value than the default of 30 minutes, consider how long your workloads will need an outbound connection. Also consider the default timeout value for a *Standard* SKU load balancer used outside of AKS is 4 minutes. An *IdleTimeoutInMinutes* value that more accurately reflects your specific AKS workload can help decrease SNAT exhaustion caused by tying up connections no longer being used.
 
 ## Restrict access to specific IP ranges
 
@@ -237,9 +281,12 @@ Learn more about Kubernetes services at the [Kubernetes services documentation][
 [azure-lb-comparison]: ../load-balancer/concepts-limitations.md#skus
 [azure-lb-outbound-rules]: ../load-balancer/load-balancer-outbound-rules-overview.md#snatports
 [azure-lb-outbound-connections]: ../load-balancer/load-balancer-outbound-connections.md#snat
+[azure-lb-outbound-preallocatedports]: ../load-balancer/load-balancer-outbound-connections.md#preallocatedports
+[azure-lb-outbound-rules-overview]: ../load-balancer/load-balancer-outbound-rules-overview.md
 [install-azure-cli]: /cli/azure/install-azure-cli
 [internal-lb-yaml]: internal-lb.md#create-an-internal-load-balancer
 [kubernetes-concepts]: concepts-clusters-workloads.md
 [use-kubenet]: configure-kubenet.md
 [az-extension-add]: /cli/azure/extension#az-extension-add
 [az-extension-update]: /cli/azure/extension#az-extension-update
+[calculate-required-quota]: #required-quota-for-customizing-allocatedoutboundports