Skip to content

Commit 66c9a91

Browse files
schaffererinschaffererin
committed
AKS - Rearchitect long docs: Configure Azure CNI networking in AKS
1 parent 42edd13 commit 66c9a91

File tree

2 files changed

+69
-106
lines changed

2 files changed

+69
-106
lines changed

‎articles/aks/configure-azure-cni-dynamic-ip-allocation.md

Lines changed: 57 additions & 95 deletions
Original file line numberDiff line numberDiff line change
@@ -9,52 +9,40 @@ ms.custom: references_regions, devx-track-azurecli
99

1010
## Configure Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in Azure Kubernetes Service (AKS)
1111

12-
A drawback with the traditional CNI is the exhaustion of pod IP addresses as the AKS cluster grows, resulting in the need to rebuild the entire cluster in a bigger subnet. The new dynamic IP allocation capability in Azure CNI solves this problem by allocating pod IPs from a subnet separate from the subnet hosting the AKS cluster. It offers the following benefits:
12+
A drawback with the traditional CNI is the exhaustion of pod IP addresses as the AKS cluster grows, which results in the need to rebuild your entire cluster in a bigger subnet. The new dynamic IP allocation capability in Azure CNI solves this problem by allocating pod IPs from a subnet separate from the subnet hosting the AKS cluster.
1313

14-
* **Better IP utilization**: IPs are dynamically allocated to cluster Pods from the Pod subnet. This leads to better utilization of IPs in the cluster compared to the traditional CNI solution, which does static allocation of IPs for every node.
14+
It offers the following benefits:
1515

16-
* **Scalable and flexible**: Node and pod subnets can be scaled independently. A single pod subnet can be shared across multiple node pools of a cluster or across multiple AKS clusters deployed in the same VNet. You can also configure a separate pod subnet for a node pool.
17-
18-
* **High performance**: Since pod are assigned VNet IPs, they have direct connectivity to other cluster pod and resources in the VNet. The solution supports very large clusters without any degradation in performance.
19-
20-
* **Separate VNet policies for pods**: Since pods have a separate subnet, you can configure separate VNet policies for them that are different from node policies. This enables many useful scenarios such as allowing internet connectivity only for pods and not for nodes, fixing the source IP for pod in a node pool using a VNet Network NAT, and using NSGs to filter traffic between node pools.
21-
22-
* **Kubernetes network policies**: Both the Azure Network Policies and Calico work with this new solution.
16+
* **Better IP utilization**: IPs are dynamically allocated to cluster Pods from the Pod subnet. This leads to better utilization of IPs in the cluster compared to the traditional CNI solution, which does static allocation of IPs for every node.
17+
* **Scalable and flexible**: Node and pod subnets can be scaled independently. A single pod subnet can be shared across multiple node pools of a cluster or across multiple AKS clusters deployed in the same VNet. You can also configure a separate pod subnet for a node pool.
18+
* **High performance**: Since pod are assigned VNet IPs, they have direct connectivity to other cluster pod and resources in the VNet. The solution supports very large clusters without any degradation in performance.
19+
* **Separate VNet policies for pods**: Since pods have a separate subnet, you can configure separate VNet policies for them that are different from node policies. This enables many useful scenarios such as allowing internet connectivity only for pods and not for nodes, fixing the source IP for pod in a node pool using a VNet Network NAT, and using NSGs to filter traffic between node pools.
20+
* **Kubernetes network policies**: Both the Azure Network Policies and Calico work with this new solution.
2321

2422
This article shows you how to use Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in AKS.
2523

2624
## Prerequisites
2725

28-
* The virtual network for the AKS cluster must allow outbound internet connectivity.
29-
* AKS clusters may not use `169.254.0.0/16`, `172.30.0.0/16`, `172.31.0.0/16`, or `192.0.2.0/24` for the Kubernetes service address range, pod address range, or cluster virtual network address range.
30-
* The cluster identity used by the AKS cluster must have at least [Network Contributor](../role-based-access-control/built-in-roles.md#network-contributor) permissions on the subnet within your virtual network. If you wish to define a [custom role](../role-based-access-control/custom-roles.md) instead of using the built-in Network Contributor role, the following permissions are required:
31-
* `Microsoft.Network/virtualNetworks/subnets/join/action`
32-
* `Microsoft.Network/virtualNetworks/subnets/read`
33-
* The subnet assigned to the AKS node pool cannot be a [delegated subnet](../virtual-network/subnet-delegation-overview.md).
34-
* AKS doesn't apply Network Security Groups (NSGs) to its subnet and will not modify any of the NSGs associated with that subnet. If you provide your own subnet and add NSGs associated with that subnet, you must ensure the security rules in the NSGs allow traffic within the node CIDR range. For more details, see [Network security groups][aks-network-nsg].
35-
36-
### Additional prerequisites
37-
3826
> [!NOTE]
39-
> When using dynamic allocation of IPs, exposing an application as a Private Link Service using a Kubernetes Load Balancer Service is not supported.
40-
41-
The [prerequisites][prerequisites] already listed for Azure CNI still apply, but there are a few additional limitations:
27+
> When using dynamic allocation of IPs, exposing an application as a Private Link Service using a Kubernetes Load Balancer Service isn't supported.
4228
29+
* Review the [prerequisites](/configure-azure-cni.md#prerequisites) for configuring basic Azure CNI networking in AKS, as the same prerequisites apply to this article.
30+
* Review the [deployment parameters](/configure-azure-cni.md#deployment-parameters) for configuring basic Azure CNI networking in AKS, as the same parameters apply.
4331
* Only linux node clusters and node pools are supported.
44-
* AKS Engine and DIY clusters are not supported.
32+
* AKS Engine and DIY clusters aren't supported.
4533
* Azure CLI version `2.37.0` or later.
4634

47-
### Planning IP addressing
35+
## Plan IP addressing
4836

49-
When using this feature, planning is much simpler. Since the nodes and pods scale independently, their address spaces can also be planned separately. Since pod subnets can be configured to the granularity of a node pool, customers can always add a new subnet when they add a node pool. The system pods in a cluster/node pool also receive IPs from the pod subnet, so this behavior needs to be accounted for.
37+
Planning your IP addressing is much simpler with this feature. Since the nodes and pods scale independently, their address spaces can also be planned separately. Since pod subnets can be configured to the granularity of a node pool, you can always add a new subnet when you add a node pool. The system pods in a cluster/node pool also receive IPs from the pod subnet, so this behavior needs to be accounted for.
5038

5139
IPs are allocated to nodes in batches of 16. Pod subnet IP allocation should be planned with a minimum of 16 IPs per node in the cluster; nodes will request 16 IPs on startup and will request another batch of 16 any time there are <8 IPs unallocated in their allotment.
5240

5341
The planning of IPs for Kubernetes services and Docker bridge remain unchanged.
5442

55-
### Maximum pods per node in a cluster with dynamic allocation of IPs and enhanced subnet support
43+
## Maximum pods per node in a cluster with dynamic allocation of IPs and enhanced subnet support
5644

57-
The pods per node values when using Azure CNI with dynamic allocation of IPs have changed slightly from the traditional CNI behavior:
45+
The pods per node values when using Azure CNI with dynamic allocation of IPs slightly differ from the traditional CNI behavior:
5846

5947
|CNI|Default|Configurable at deployment|
6048
|--| :--: |--|
@@ -63,74 +51,74 @@ The pods per node values when using Azure CNI with dynamic allocation of IPs hav
6351

6452
All other guidance related to configuring the maximum pods per node remains the same.
6553

66-
### Additional deployment parameters
54+
## Deployment parameters
6755

68-
The deployment parameters described above are all still valid, with one exception:
56+
The [deployment parameters](/configure-azure-cni.md#deployment-parameters) for configuring basic Azure CNI networking in AKS are all valid, with two exceptions:
6957

7058
* The **subnet** parameter now refers to the subnet related to the cluster's nodes.
7159
* An additional parameter **pod subnet** is used to specify the subnet whose IP addresses will be dynamically allocated to pods.
7260

73-
### Configure networking - CLI with dynamic allocation of IPs and enhanced subnet support
61+
## Configure networking with dynamic allocation of IPs and enhanced subnet support - Azure CLI
7462

75-
Using dynamic allocation of IPs and enhanced subnet support in your cluster is similar to the default method for configuring a cluster Azure CNI. The following example walks through creating a new virtual network with a subnet for nodes and a subnet for pods, and creating a cluster that uses Azure CNI with dynamic allocation of IPs and enhanced subnet support. Be sure to replace variables such as `$subscription` with your own values:
63+
Using dynamic allocation of IPs and enhanced subnet support in your cluster is similar to the default method for configuring a cluster Azure CNI. The following example walks through creating a new virtual network with a subnet for nodes and a subnet for pods, and creating a cluster that uses Azure CNI with dynamic allocation of IPs and enhanced subnet support. Be sure to replace variables such as `$subscription` with your own values.
7664

77-
First, create the virtual network with two subnets:
65+
Create the virtual network with two subnets.
7866

79-
```azurecli-interactive
80-
resourceGroup="myResourceGroup"
81-
vnet="myVirtualNetwork"
82-
location="westcentralus"
67+
```azurecli-interactive
68+
resourceGroup="myResourceGroup"
69+
vnet="myVirtualNetwork"
70+
location="westcentralus"
8371

84-
# Create the resource group
85-
az group create --name $resourceGroup --location $location
72+
# Create the resource group
73+
az group create --name $resourceGroup --location $location
8674

87-
# Create our two subnet network
88-
az network vnet create -g $resourceGroup --location $location --name $vnet --address-prefixes 10.0.0.0/8 -o none
89-
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name nodesubnet --address-prefixes 10.240.0.0/16 -o none
90-
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name podsubnet --address-prefixes 10.241.0.0/16 -o none
91-
```
75+
# Create our two subnet network
76+
az network vnet create -g $resourceGroup --location $location --name $vnet --address-prefixes 10.0.0.0/8 -o none
77+
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name nodesubnet --address-prefixes 10.240.0.0/16 -o none
78+
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name podsubnet --address-prefixes 10.241.0.0/16 -o none
79+
```
9280

93-
Then, create the cluster, referencing the node subnet using `--vnet-subnet-id` and the pod subnet using `--pod-subnet-id`:
81+
Create the cluster, referencing the node subnet using `--vnet-subnet-id` and the pod subnet using `--pod-subnet-id`.
9482

95-
```azurecli-interactive
96-
clusterName="myAKSCluster"
97-
subscription="aaaaaaa-aaaaa-aaaaaa-aaaa"
83+
```azurecli-interactive
84+
clusterName="myAKSCluster"
85+
subscription="aaaaaaa-aaaaa-aaaaaa-aaaa"
9886

99-
az aks create -n $clusterName -g $resourceGroup -l $location \
100-
--max-pods 250 \
101-
--node-count 2 \
102-
--network-plugin azure \
103-
--vnet-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/nodesubnet \
104-
--pod-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/podsubnet
105-
```
87+
az aks create -n $clusterName -g $resourceGroup -l $location \
88+
--max-pods 250 \
89+
--node-count 2 \
90+
--network-plugin azure \
91+
--vnet-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/nodesubnet \
92+
--pod-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/podsubnet
93+
```
10694

107-
#### Adding node pool
95+
### Adding node pool
10896

10997
When adding node pool, reference the node subnet using `--vnet-subnet-id` and the pod subnet using `--pod-subnet-id`. The following example creates two new subnets that are then referenced in the creation of a new node pool:
11098

111-
```azurecli-interactive
112-
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name node2subnet --address-prefixes 10.242.0.0/16 -o none
113-
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name pod2subnet --address-prefixes 10.243.0.0/16 -o none
99+
```azurecli-interactive
100+
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name node2subnet --address-prefixes 10.242.0.0/16 -o none
101+
az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name pod2subnet --address-prefixes 10.243.0.0/16 -o none
114102

115-
az aks nodepool add --cluster-name $clusterName -g $resourceGroup -n newnodepool \
116-
--max-pods 250 \
117-
--node-count 2 \
118-
--vnet-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/node2subnet \
119-
--pod-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/pod2subnet \
120-
--no-wait
121-
```
103+
az aks nodepool add --cluster-name $clusterName -g $resourceGroup -n newnodepool \
104+
--max-pods 250 \
105+
--node-count 2 \
106+
--vnet-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/node2subnet \
107+
--pod-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/pod2subnet \
108+
--no-wait
109+
```
122110

123-
### Dynamic allocation of IP addresses and enhanced subnet support FAQs
111+
## Dynamic allocation of IP addresses and enhanced subnet support FAQs
124112

125-
* *Can I assign multiple pod subnets to a cluster/node pool?*
113+
* **Can I assign multiple pod subnets to a cluster/node pool?**
126114

127115
Only one subnet can be assigned to a cluster or node pool. However, multiple clusters or node pools can share a single subnet.
128116

129-
* *Can I assign Pod subnets from a different VNet altogether?*
117+
* **Can I assign Pod subnets from a different VNet altogether?**
130118

131119
No, the pod subnet should be from the same VNet as the cluster.
132120

133-
* *Can some node pools in a cluster use the traditional CNI while others use the new CNI?*
121+
* **Can some node pools in a cluster use the traditional CNI while others use the new CNI?**
134122

135123
The entire cluster should use only one type of CNI.
136124

@@ -147,35 +135,9 @@ Learn more about networking in AKS in the following articles:
147135
* [Create an ingress controller with a dynamic public IP and configure Let's Encrypt to automatically generate TLS certificates][aks-ingress-tls]
148136
* [Create an ingress controller with a static public IP and configure Let's Encrypt to automatically generate TLS certificates][aks-ingress-static-tls]
149137

150-
<!-- IMAGES -->
151-
[advanced-networking-diagram-01]: ./media/networking-overview/advanced-networking-diagram-01.png
152-
[portal-01-networking-advanced]: ./media/networking-overview/portal-01-networking-advanced.png
153-
154-
<!-- LINKS - External -->
155-
[services]: https://kubernetes.io/docs/concepts/services-networking/service/
156-
[portal]: https://portal.azure.com
157-
[cni-networking]: https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md
158-
[kubenet]: concepts-network.md#kubenet-basic-networking
159-
[github]: https://raw.githubusercontent.com/microsoft/Docker-Provider/ci_prod/kubernetes/container-azm-ms-agentconfig.yaml
160-
161138
<!-- LINKS - Internal -->
162-
[az-aks-create]: /cli/azure/aks#az_aks_create
163-
[aks-ssh]: ssh.md
164-
[ManagedClusterAgentPoolProfile]: /azure/templates/microsoft.containerservice/managedclusters#managedclusteragentpoolprofile-object
165-
[aks-network-concepts]: concepts-network.md
166-
[aks-network-nsg]: concepts-network.md#network-security-groups
167139
[aks-ingress-basic]: ingress-basic.md
168140
[aks-ingress-tls]: ingress-tls.md
169141
[aks-ingress-static-tls]: ingress-static-ip.md
170142
[aks-http-app-routing]: http-application-routing.md
171143
[aks-ingress-internal]: ingress-internal-ip.md
172-
[az-extension-add]: /cli/azure/extension#az_extension_add
173-
[az-extension-update]: /cli/azure/extension#az_extension_update
174-
[az-feature-register]: /cli/azure/feature#az_feature_register
175-
[az-feature-list]: /cli/azure/feature#az_feature_list
176-
[az-provider-register]: /cli/azure/provider#az_provider_register
177-
[network-policy]: use-network-policies.md
178-
[nodepool-upgrade]: use-multiple-node-pools.md#upgrade-a-node-pool
179-
[network-comparisons]: concepts-network.md#compare-network-models
180-
[system-node-pools]: use-system-pools.md
181-
[prerequisites]: configure-azure-cni.md#prerequisites

0 commit comments

Comments
 (0)