Merge pull request #223722 from phealy/pahealy/doc-updates

Add AKS features

Author: jborsecnik

articles/aks/TOC.yml

@@ -248,6 +248,8 @@
               href: use-cvm.md
             - name: Use system node pools
               href: use-system-pools.md
+            - name: Use instance-level public IP addresses
+              href: use-node-public-ips.md
             - name: Use WebAssembly System Interface (WASI) node pools
               href: use-wasi-node-pools.md
             - name: Start/stop node pools
@@ -388,8 +390,10 @@
           items:
             - name: Restrict and control cluster egress traffic
               href: limit-egress-traffic.md
-            - name: Use a user defined route for egress
+            - name: Configure outbound type for AKS
               href: egress-outboundtype.md
+            - name: Use a user defined route for egress
+              href: egress-udr.md
             - name: Managed NAT Gateway
               href: nat-gateway.md
             - name: Use an HTTP proxy

articles/aks/api-server-authorized-ip-ranges.md

@@ -26,7 +26,7 @@ The *API server authorized IP ranges* feature has the following limitations:
 
 - The *API server authorized IP ranges* feature was moved out of preview in October 2019. For clusters created after the feature was moved out of preview, this feature is only supported on the *Standard* SKU load balancer. Any existing clusters on the *Basic* SKU load balancer with the *API server authorized IP ranges* feature enabled will continue to work as is. However, these clusters cannot be migrated to a *Standard* SKU load balancer. Existing clusters will continue to work if the Kubernetes version and control plane are upgraded.
 - The *API server authorized IP ranges* feature isn't supported on private clusters.
-- When using this feature with clusters that use [Node Public IP](use-multiple-node-pools.md#assign-a-public-ip-per-node-for-your-node-pools), the node pools using Node Public IP must use public IP prefixes. The public IP prefixes must be added as authorized ranges.
+- When using this feature with clusters that use [Node Public IP](use-node-public-ips.md), the node pools using Node Public IP must use public IP prefixes. The public IP prefixes must be added as authorized ranges.
 
 ## Overview of API server authorized IP ranges
 
@@ -38,7 +38,7 @@ For more information about the API server and other cluster components, see [Kub
 
 ## Create an AKS cluster with API server authorized IP ranges enabled
 
-Create a cluster using the [az aks create][az-aks-create] and specify the *`--api-server-authorized-ip-ranges`* parameter to provide a list of authorized IP address ranges. These IP address ranges are usually address ranges used by your on-premises networks or public IPs. When you specify a CIDR range, start with the first IP address in the range. For example, *137.117.106.90/29* is a valid range, but make sure you specify the first IP address in the range, such as *137.117.106.88/29*.
+Create a cluster using the [`az aks create`][az-aks-create] and specify the *`--api-server-authorized-ip-ranges`* parameter to provide a list of authorized IP address ranges. These IP address ranges are usually address ranges used by your on-premises networks or public IPs. When you specify a CIDR range, start with the first IP address in the range. For example, *137.117.106.90/29* is a valid range, but make sure you specify the first IP address in the range, such as *137.117.106.88/29*.
 
 > [!IMPORTANT]
 > By default, your cluster uses the [Standard SKU load balancer][standard-sku-lb] which you can use to configure the outbound gateway. When you enable API server authorized IP ranges during cluster creation, the public IP for your cluster is also allowed by default in addition to the ranges you specify. If you specify *""* or no value for *`--api-server-authorized-ip-ranges`*, API server authorized IP ranges will be disabled. Note that if you're using PowerShell, use *`--api-server-authorized-ip-ranges=""`* (with equals sign) to avoid any parsing issues.
@@ -105,7 +105,7 @@ az aks create \
 
 ## Update a cluster's API server authorized IP ranges
 
-To update the API server authorized IP ranges on an existing cluster, use [az aks update][az-aks-update] command and use the *`--api-server-authorized-ip-ranges`*, *`--load-balancer-outbound-ip-prefixes`*, *`--load-balancer-outbound-ips`*, or *`--load-balancer-outbound-ip-prefixes`* parameters.
+To update the API server authorized IP ranges on an existing cluster, use [`az aks update`][az-aks-update] command and use the *`--api-server-authorized-ip-ranges`*, *`--load-balancer-outbound-ip-prefixes`*, *`--load-balancer-outbound-ips`*, or *`--load-balancer-outbound-ip-prefixes`* parameters.
 
 The following example updates API server authorized IP ranges on the cluster named *myAKSCluster* in the resource group named *myResourceGroup*. The IP address range to authorize is *73.140.245.0/24*:
 
@@ -120,7 +120,7 @@ You can also use *0.0.0.0/32* when specifying the *`--api-server-authorized-ip-r
 
 ## Disable authorized IP ranges
 
-To disable authorized IP ranges, use [az aks update][az-aks-update] and specify an empty range to disable API server authorized IP ranges. For example:
+To disable authorized IP ranges, use [`az aks update`][az-aks-update] and specify an empty range to disable API server authorized IP ranges. For example:
 
 ```azurecli-interactive
 az aks update \
@@ -137,7 +137,7 @@ az aks update \
 
 ## Find existing authorized IP ranges
 
-To find IP ranges that have been authorized, use [az aks show][az-aks-show] and specify the cluster's name and resource group. For example:
+To find IP ranges that have been authorized, use [`az aks show`][az-aks-show] and specify the cluster's name and resource group. For example:
 
 ```azurecli-interactive
 az aks show \

articles/aks/azure-cni-overlay.md

@@ -84,9 +84,8 @@ Use the traditional VNet option when:
 
 ## Limitations with Azure CNI Overlay
 
-The overlay solution has the following limitations today
+The overlay solution has the following limitations:
 
-* You can't deploy multiple overlay clusters on the same subnet.
 * Overlay can be enabled only for new clusters. Existing (already deployed) clusters can't be configured to use overlay.
 * You can't use Application Gateway as an Ingress Controller (AGIC) for an overlay cluster.
 

articles/aks/configure-azure-cni.md

@@ -77,7 +77,7 @@ A minimum value for maximum pods per node is enforced to guarantee space for sys
 > [!NOTE]
 > The minimum value in the table above is strictly enforced by the AKS service. You can not set a maxPods value lower than the minimum shown as doing so can prevent the cluster from starting.
 
-* **Azure CLI**: Specify the `--max-pods` argument when you deploy a cluster with the [az aks create][az-aks-create] command. The maximum value is 250.
+* **Azure CLI**: Specify the `--max-pods` argument when you deploy a cluster with the [`az aks create`][az-aks-create] command. The maximum value is 250.
 * **Resource Manager template**: Specify the `maxPods` property in the [ManagedClusterAgentPoolProfile] object when you deploy a cluster with a Resource Manager template. The maximum value is 250.
 * **Azure portal**: Change the `Max pods per node` field in the node pool settings when creating a cluster or adding a new node pool.
 
@@ -123,7 +123,7 @@ $ az network vnet subnet list \
 /subscriptions/<guid>/resourceGroups/myVnet/providers/Microsoft.Network/virtualNetworks/myVnet/subnets/default
 ```
 
-Use the [az aks create][az-aks-create] command with the `--network-plugin azure` argument to create a cluster with advanced networking. Update the `--vnet-subnet-id` value with the subnet ID collected in the previous step:
+Use the [`az aks create`][az-aks-create] command with the `--network-plugin azure` argument to create a cluster with advanced networking. Update the `--vnet-subnet-id` value with the subnet ID collected in the previous step:
 
 ```azurecli-interactive
 az aks create \
@@ -164,7 +164,6 @@ A drawback with the traditional CNI is the exhaustion of pod IP addresses as the
 
 The [prerequisites][prerequisites] already listed for Azure CNI still apply, but there are a few additional limitations:
 
-* Only linux node clusters and node pools are supported.
 * AKS Engine and DIY clusters are not supported.
 * Azure CLI version `2.37.0` or later.
 
@@ -248,7 +247,7 @@ az aks nodepool add --cluster-name $clusterName -g $resourceGroup  -n newnodepoo
 Azure CNI provides the capability to monitor IP subnet usage. To enable IP subnet usage monitoring, follow the steps below:
 
 ### Get the YAML file
-1.	Download or grep the file named container-azm-ms-agentconfig.yaml from [github][github].
+1.	Download or grep the file named container-azm-ms-agentconfig.yaml from [GitHub][github].
 2.	Find azure_subnet_ip_usage in integrations. Set `enabled` to `true`. 
 3.	Save the file.
 

articles/aks/egress-outboundtype.md

@@ -1,34 +1,29 @@
 ---
-title: Customize user-defined routes (UDR) in Azure Kubernetes Service (AKS)
-description: Learn how to define a custom egress route in Azure Kubernetes Service (AKS)
+title: Customize cluster egress with outbound types in Azure Kubernetes Service (AKS)
+description: Learn how to configure outbound types in Azure Kubernetes Service (AKS)
 services: container-service
 ms.topic: article
 ms.date: 06/29/2020
-
+ms.author: pahealy
+author: phealy
 
 #Customer intent: As a cluster operator, I want to define my own egress paths with user-defined routes. Since I define this up front I do not want AKS provided load balancer configurations.
 ---
 
-# Customize cluster egress with a User-Defined Route
+# Customize cluster egress with outbound types in Azure Kubernetes Service (AKS)
 
 Egress from an AKS cluster can be customized to fit specific scenarios. By default, AKS will provision a Standard SKU Load Balancer to be set up and used for egress. However, the default setup may not meet the requirements of all scenarios if public IPs are disallowed or additional hops are required for egress.
 
-This article walks through how to customize a cluster's egress route to support custom network scenarios, such as those which disallows public IPs and requires the cluster to sit behind a network virtual appliance (NVA).
-
-## Prerequisites
-* Azure CLI version 2.0.81 or greater
-* API version of `2020-01-01` or greater
-
+This article covers the various types of outbound connectivity that are available in AKS Clusters.
 
 ## Limitations
-* OutboundType can only be defined at cluster create time and can't be updated afterwards.
+* Outbound type can only be defined at cluster create time and can't be updated afterwards.
+  * Reconfiguring outbound type is now supported in preview; see below.
 * Setting `outboundType` requires AKS clusters with a `vm-set-type` of `VirtualMachineScaleSets` and `load-balancer-sku` of `Standard`.
-* Setting `outboundType` to a value of `UDR` requires a user-defined route with valid outbound connectivity for the cluster.
-* Setting `outboundType` to a value of `UDR` implies the ingress source IP routed to the load-balancer may **not match** the cluster's outgoing egress destination address.
 
 ## Overview of outbound types in AKS
 
-An AKS cluster can be customized with a unique `outboundType` of type `loadBalancer` or `userDefinedRouting`.
+An AKS cluster can be configured with three different categories of outbound type: load balancer, NAT gateway, or user-defined routing.
 
 > [!IMPORTANT]
 > Outbound type impacts only the egress traffic of your cluster. For more information, see [setting up ingress controllers](ingress-basic.md).
@@ -49,6 +44,19 @@ Below is a network topology deployed in AKS clusters by default, which use an `o
 
 ![Diagram shows ingress I P and egress I P, where the ingress I P directs traffic to a load balancer, which directs traffic to and from an internal cluster and other traffic to the egress I P, which directs traffic to the Internet, M C R, Azure required services, and the A K S Control Plane.](media/egress-outboundtype/outboundtype-lb.png)
 
+For more information, see [using a standard load balancer in AKS](load-balancer-standard.md) for more information.
+
+### Outbound type of `managedNatGateway` or `userAssignedNatGateway`
+
+If `managedNatGateway` or `userAssignedNatGateway` are selected for `outboundType`, AKS relies on [Azure Networking NAT gateway](/azure/virtual-network/nat-gateway/manage-nat-gateway) for cluster egress. 
+
+- `managedNatGateway` is used when using managed virtual networks, and tells AKS to provision a NAT gateway and attach it to the cluster subnet.
+- `userAssignedNatGateway` is used when using bring-your-own virtual networking, and requires that a NAT gateway has been provisioned before cluster creation.
+
+NAT gateway has significantly improved handling of SNAT ports when compared to Standard Load Balancer.
+
+For more information, see [using NAT Gateway with AKS](nat-gateway.md) for more information.
+
 ### Outbound type of userDefinedRouting
 
 > [!NOTE]
@@ -58,26 +66,71 @@ If `userDefinedRouting` is set, AKS won't automatically configure egress paths.
 
 The AKS cluster must be deployed into an existing virtual network with a subnet that has been previously configured because when not using standard load balancer (SLB) architecture, you must establish explicit egress. As such, this architecture requires explicitly sending egress traffic to an appliance like a firewall, gateway, proxy or to allow the Network Address Translation (NAT) to be done by a public IP assigned to the standard load balancer or appliance.
 
-#### Load balancer creation with userDefinedRouting
+For more information, see [configuring cluster egress via user-defined routing](egress-udr.md) for more information.
 
-AKS clusters with an outbound type of UDR receive a standard load balancer (SLB) only when the first Kubernetes service of type 'loadBalancer' is deployed. The load balancer is configured with a public IP address for *inbound* requests and a backend pool for *inbound* requests. Inbound rules are configured by the Azure cloud provider, but **no outbound public IP address or outbound rules** are configured as a result of having an outbound type of UDR. Your UDR will still be the only source for egress traffic.
+## Updating `outboundType` after cluster creation (PREVIEW)
 
-Azure load balancers [don't incur a charge until a rule is placed](https://azure.microsoft.com/pricing/details/load-balancer/).
+Changing the outbound type after cluster creation will deploy or remove resources as required to put the cluster into the new egress configuration.
 
-## Deploy a cluster with outbound type of UDR and Azure Firewall
+Migration is only supported between `loadBalancer`, `managedNATGateway` (if using a managed virtual network), and `userDefinedNATGateway` (if using a custom virtual network).
 
-To illustrate the application of a cluster with outbound type using a user-defined route, a cluster can be configured on a virtual network with an Azure Firewall on its own subnet. See this example on the [restrict egress traffic with Azure firewall example](limit-egress-traffic.md#restrict-egress-traffic-using-azure-firewall).
+> [!WARNING]
+> Changing the outbound type on a cluster is disruptive to network connectivity and will result in a change of the cluster's egress IP address. If any firewall rules have been configured to restrict traffic from the cluster, they will need to be updated to match the new egress IP address.
 
-> [!IMPORTANT]
-> Outbound type of UDR requires there is a route for 0.0.0.0/0 and next hop destination of NVA (Network Virtual Appliance) in the route table.
-> The route table already has a default 0.0.0.0/0 to Internet, without a Public IP to SNAT just adding this route will not provide you egress. AKS will validate that you don't create a 0.0.0.0/0 route pointing to the Internet but instead to NVA or gateway, etc.
-> When using an outbound type of UDR, a load balancer public IP address for **inbound requests** is not created unless a service of type *loadbalancer* is configured. A public IP address for **outbound requests** is never created by AKS if an outbound type of UDR is set.
+[!INCLUDE [preview features callout](includes/preview/preview-callout.md)]
 
-## Next steps
+### Install the aks-preview Azure CLI extension
+
+`aks-preview` version 0.5.113 is required.
+
+To install the `aks-preview` extension, run the following command:
+
+```azurecli
+az extension add --name aks-preview
+```
 
-See [Azure networking UDR overview](../virtual-network/virtual-networks-udr-overview.md).
+Run the following command to update to the latest version of the extension released:
+
+```azurecli
+az extension update --name aks-preview
+```
+
+### Register the 'AKS-OutBoundTypeMigrationPreview' feature flag
+
+Register the `AKS-OutBoundTypeMigrationPreview` feature flag by using the [az feature register][az-feature-register] command, as shown in the following example:
+
+```azurecli-interactive
+az feature register --namespace "Microsoft.ContainerService" --name "AKS-OutBoundTypeMigrationPreview"
+```
+
+It takes a few minutes for the status to show *Registered*. Verify the registration status by using the [az feature show][az-feature-show] command:
+
+```azurecli-interactive
+az feature show --namespace "Microsoft.ContainerService" --name "AKS-OutBoundTypeMigrationPreview"
+```
+
+When the status reflects *Registered*, refresh the registration of the *Microsoft.ContainerService* resource provider by using the [az provider register][az-provider-register] command:
+
+```azurecli-interactive
+az provider register --namespace Microsoft.ContainerService
+```
+
+### Update a cluster to use a new outbound type
+
+Run the following command to change a cluster's outbound configuration:
+
+```azurecli-interactive
+az aks update -g <resourceGroup> -n <clusterName> --outbound-type <loadBalancer|managedNATGateway|userAssignedNATGateway>
+```
+
+## Next steps
 
-See [how to create, change, or delete a route table](../virtual-network/manage-route-table.md).
+- [Configure standard load balancing in an AKS cluster](load-balancer-standard.md)
+- [Configure NAT gateway in an AKS cluster](nat-gateway.md)
+- [Configure user-defined routing in an AKS cluster](egress-udr.md)
+- [NAT gateway documentation](/azure/aks/nat-gateway)
+- [Azure networking UDR overview](../virtual-network/virtual-networks-udr-overview.md).
+- [Manage route tables](../virtual-network/manage-route-table.md).
 
 <!-- LINKS - internal -->
 [az-aks-get-credentials]: /cli/azure/aks#az_aks_get_credentials