Skip to content

Commit 42edd13

Browse files
schaffererinschaffererin
committed
Rearchitect long AKS docs
1 parent 1694e15 commit 42edd13

File tree

3 files changed

+187
-120
lines changed

3 files changed

+187
-120
lines changed

‎articles/aks/TOC.yml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -374,9 +374,11 @@
374374
href: use-byo-cni.md
375375
- name: Use Azure CNI Powered by Cilium (Preview)
376376
href: azure-cni-powered-by-cilium.md
377-
- name: Use Azure-CNI
377+
- name: Use Azure CNI
378378
href: configure-azure-cni.md
379-
- name: Use Azure-CNI Overlay (Preview)
379+
- name: Use Azure CNI for dynamic IP allocation and enhanced subnet support
380+
href: configure-azure-cni-dynamic-ip-allocation.md
381+
- name: Use Azure CNI Overlay (Preview)
380382
href: azure-cni-overlay.md
381383
- name: DNS
382384
items:

‎articles/aks/configure-azure-cni-dynamic-ip-allocation.md

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
---
2+
title: Configure Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in Azure Kubernetes Service (AKS)
3+
description: Learn how to configure Azure CNI (advanced) networking for dynamic allocation of IPs and enhanced subnet support in Azure Kubernetes Service (AKS)
4+
services: container-service
5+
ms.topic: article
6+
ms.date: 01/09/2023
7+
ms.custom: references_regions, devx-track-azurecli
8+
---
9+
10+
## Configure Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in Azure Kubernetes Service (AKS)
11+
12+
A drawback with the traditional CNI is the exhaustion of pod IP addresses as the AKS cluster grows, resulting in the need to rebuild the entire cluster in a bigger subnet. The new dynamic IP allocation capability in Azure CNI solves this problem by allocating pod IPs from a subnet separate from the subnet hosting the AKS cluster. It offers the following benefits:
13+
14+
* **Better IP utilization**: IPs are dynamically allocated to cluster Pods from the Pod subnet. This leads to better utilization of IPs in the cluster compared to the traditional CNI solution, which does static allocation of IPs for every node.
15+
16+
* **Scalable and flexible**: Node and pod subnets can be scaled independently. A single pod subnet can be shared across multiple node pools of a cluster or across multiple AKS clusters deployed in the same VNet. You can also configure a separate pod subnet for a node pool.
17+
18+
* **High performance**: Since pod are assigned VNet IPs, they have direct connectivity to other cluster pod and resources in the VNet. The solution supports very large clusters without any degradation in performance.
19+
20+
* **Separate VNet policies for pods**: Since pods have a separate subnet, you can configure separate VNet policies for them that are different from node policies. This enables many useful scenarios such as allowing internet connectivity only for pods and not for nodes, fixing the source IP for pod in a node pool using a VNet Network NAT, and using NSGs to filter traffic between node pools.
21+
22+
* **Kubernetes network policies**: Both the Azure Network Policies and Calico work with this new solution.
23+
24+
This article shows you how to use Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in AKS.
25+
26+
## Prerequisites
27+
28+
* The virtual network for the AKS cluster must allow outbound internet connectivity.
29+
* AKS clusters may not use `169.254.0.0/16`, `172.30.0.0/16`, `172.31.0.0/16`, or `192.0.2.0/24` for the Kubernetes service address range, pod address range, or cluster virtual network address range.
30+
* The cluster identity used by the AKS cluster must have at least [Network Contributor](../role-based-access-control/built-in-roles.md#network-contributor) permissions on the subnet within your virtual network. If you wish to define a [custom role](../role-based-access-control/custom-roles.md) instead of using the built-in Network Contributor role, the following permissions are required:
31+
* `Microsoft.Network/virtualNetworks/subnets/join/action`
32+
* `Microsoft.Network/virtualNetworks/subnets/read`
33+
* The subnet assigned to the AKS node pool cannot be a [delegated subnet](../virtual-network/subnet-delegation-overview.md).
34+
* AKS doesn't apply Network Security Groups (NSGs) to its subnet and will not modify any of the NSGs associated with that subnet. If you provide your own subnet and add NSGs associated with that subnet, you must ensure the security rules in the NSGs allow traffic within the node CIDR range. For more details, see [Network security groups][aks-network-nsg].
35+
36+
### Additional prerequisites
37+
38+
> [!NOTE]
39+
> When using dynamic allocation of IPs, exposing an application as a Private Link Service using a Kubernetes Load Balancer Service is not supported.
40+
41+
The [prerequisites][prerequisites] already listed for Azure CNI still apply, but there are a few additional limitations:
42+
43+
* Only linux node clusters and node pools are supported.
44+
* AKS Engine and DIY clusters are not supported.
45+
* Azure CLI version `2.37.0` or later.
46+
47+
### Planning IP addressing
48+
49+
When using this feature, planning is much simpler. Since the nodes and pods scale independently, their address spaces can also be planned separately. Since pod subnets can be configured to the granularity of a node pool, customers can always add a new subnet when they add a node pool. The system pods in a cluster/node pool also receive IPs from the pod subnet, so this behavior needs to be accounted for.
50+
51+
IPs are allocated to nodes in batches of 16. Pod subnet IP allocation should be planned with a minimum of 16 IPs per node in the cluster; nodes will request 16 IPs on startup and will request another batch of 16 any time there are <8 IPs unallocated in their allotment.
52+
53+
The planning of IPs for Kubernetes services and Docker bridge remain unchanged.
54+
55+
### Maximum pods per node in a cluster with dynamic allocation of IPs and enhanced subnet support
56+
57+
The pods per node values when using Azure CNI with dynamic allocation of IPs have changed slightly from the traditional CNI behavior:
58+
59+
|CNI|Default|Configurable at deployment|
60+
|--| :--: |--|
61+
|Traditional Azure CNI|30|Yes (up to 250)|
62+
|Azure CNI with dynamic allocation of IPs|250|Yes (up to 250)|
63+
64+
All other guidance related to configuring the maximum pods per node remains the same.
65+
66+
### Additional deployment parameters
67+
68+
The deployment parameters described above are all still valid, with one exception:
69+
70+
* The **subnet** parameter now refers to the subnet related to the cluster's nodes.
71+
* An additional parameter **pod subnet** is used to specify the subnet whose IP addresses will be dynamically allocated to pods.
72+
73+
### Configure networking - CLI with dynamic allocation of IPs and enhanced subnet support
74+
75+
Using dynamic allocation of IPs and enhanced subnet support in your cluster is similar to the default method for configuring a cluster Azure CNI. The following example walks through creating a new virtual network with a subnet for nodes and a subnet for pods, and creating a cluster that uses Azure CNI with dynamic allocation of IPs and enhanced subnet support. Be sure to replace variables such as `$subscription` with your own values:
76+
77+
First, create the virtual network with two subnets:
78+
79+
```azurecli-interactive
80+
resourceGroup="myResourceGroup"
81+
vnet="myVirtualNetwork"
82+
location="westcentralus"
83+
84+
# Create the resource group
85+
az group create --name $resourceGroup --location $location
86+
87+
# Create our two subnet network
88+
az network vnet create -g $resourceGroup --location $location --name $vnet --address-prefixes 10.0.0.0/8 -o none
89+
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name nodesubnet --address-prefixes 10.240.0.0/16 -o none
90+
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name podsubnet --address-prefixes 10.241.0.0/16 -o none
91+
```
92+
93+
Then, create the cluster, referencing the node subnet using `--vnet-subnet-id` and the pod subnet using `--pod-subnet-id`:
94+
95+
```azurecli-interactive
96+
clusterName="myAKSCluster"
97+
subscription="aaaaaaa-aaaaa-aaaaaa-aaaa"
98+
99+
az aks create -n $clusterName -g $resourceGroup -l $location \
100+
--max-pods 250 \
101+
--node-count 2 \
102+
--network-plugin azure \
103+
--vnet-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/nodesubnet \
104+
--pod-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/podsubnet
105+
```
106+
107+
#### Adding node pool
108+
109+
When adding node pool, reference the node subnet using `--vnet-subnet-id` and the pod subnet using `--pod-subnet-id`. The following example creates two new subnets that are then referenced in the creation of a new node pool:
110+
111+
```azurecli-interactive
112+
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name node2subnet --address-prefixes 10.242.0.0/16 -o none
113+
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name pod2subnet --address-prefixes 10.243.0.0/16 -o none
114+
115+
az aks nodepool add --cluster-name $clusterName -g $resourceGroup -n newnodepool \
116+
--max-pods 250 \
117+
--node-count 2 \
118+
--vnet-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/node2subnet \
119+
--pod-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/pod2subnet \
120+
--no-wait
121+
```
122+
123+
### Dynamic allocation of IP addresses and enhanced subnet support FAQs
124+
125+
* *Can I assign multiple pod subnets to a cluster/node pool?*
126+
127+
Only one subnet can be assigned to a cluster or node pool. However, multiple clusters or node pools can share a single subnet.
128+
129+
* *Can I assign Pod subnets from a different VNet altogether?*
130+
131+
No, the pod subnet should be from the same VNet as the cluster.
132+
133+
* *Can some node pools in a cluster use the traditional CNI while others use the new CNI?*
134+
135+
The entire cluster should use only one type of CNI.
136+
137+
## Next steps
138+
139+
Learn more about networking in AKS in the following articles:
140+
141+
* [Use a static IP address with the Azure Kubernetes Service (AKS) load balancer](static-ip.md)
142+
* [Use an internal load balancer with Azure Kubernetes Service (AKS)](internal-lb.md)
143+
144+
* [Create a basic ingress controller with external network connectivity][aks-ingress-basic]
145+
* [Enable the HTTP application routing add-on][aks-http-app-routing]
146+
* [Create an ingress controller that uses an internal, private network and IP address][aks-ingress-internal]
147+
* [Create an ingress controller with a dynamic public IP and configure Let's Encrypt to automatically generate TLS certificates][aks-ingress-tls]
148+
* [Create an ingress controller with a static public IP and configure Let's Encrypt to automatically generate TLS certificates][aks-ingress-static-tls]
149+
150+
<!-- IMAGES -->
151+
[advanced-networking-diagram-01]: ./media/networking-overview/advanced-networking-diagram-01.png
152+
[portal-01-networking-advanced]: ./media/networking-overview/portal-01-networking-advanced.png
153+
154+
<!-- LINKS - External -->
155+
[services]: https://kubernetes.io/docs/concepts/services-networking/service/
156+
[portal]: https://portal.azure.com
157+
[cni-networking]: https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md
158+
[kubenet]: concepts-network.md#kubenet-basic-networking
159+
[github]: https://raw.githubusercontent.com/microsoft/Docker-Provider/ci_prod/kubernetes/container-azm-ms-agentconfig.yaml
160+
161+
<!-- LINKS - Internal -->
162+
[az-aks-create]: /cli/azure/aks#az_aks_create
163+
[aks-ssh]: ssh.md
164+
[ManagedClusterAgentPoolProfile]: /azure/templates/microsoft.containerservice/managedclusters#managedclusteragentpoolprofile-object
165+
[aks-network-concepts]: concepts-network.md
166+
[aks-network-nsg]: concepts-network.md#network-security-groups
167+
[aks-ingress-basic]: ingress-basic.md
168+
[aks-ingress-tls]: ingress-tls.md
169+
[aks-ingress-static-tls]: ingress-static-ip.md
170+
[aks-http-app-routing]: http-application-routing.md
171+
[aks-ingress-internal]: ingress-internal-ip.md
172+
[az-extension-add]: /cli/azure/extension#az_extension_add
173+
[az-extension-update]: /cli/azure/extension#az_extension_update
174+
[az-feature-register]: /cli/azure/feature#az_feature_register
175+
[az-feature-list]: /cli/azure/feature#az_feature_list
176+
[az-provider-register]: /cli/azure/provider#az_provider_register
177+
[network-policy]: use-network-policies.md
178+
[nodepool-upgrade]: use-multiple-node-pools.md#upgrade-a-node-pool
179+
[network-comparisons]: concepts-network.md#compare-network-models
180+
[system-node-pools]: use-system-pools.md
181+
[prerequisites]: configure-azure-cni.md#prerequisites

0 commit comments

Comments
 (0)