Skip to content

Commit 86fd101

Browse files
AnnaMHuffAnnaMHuff
authored
Merge pull request #189830 from Nickomang/aks-nodepool-resize-workaround
Added resize node pool article
2 parents dd80c7b + b691386 commit 86fd101

File tree

6 files changed

+232
-97
lines changed

6 files changed

+232
-97
lines changed

‎articles/aks/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,8 @@
217217
href: use-wasi-node-pools.md
218218
- name: Start/stop node pools (preview)
219219
href: start-stop-nodepools.md
220+
- name: Resize node pools
221+
href: resize-node-pool.md
220222
- name: Deploy AKS with Terraform
221223
href: /azure/developer/terraform/create-k8s-cluster-with-tf-and-aks
222224
maintainContext: true

‎articles/aks/media/resize-node-pool/node-pool-both.png

36.5 KB
Loading

‎articles/aks/media/resize-node-pool/node-pool-ds2.png

24.7 KB
Loading

‎articles/aks/media/resize-node-pool/node-pool-ds3.png

25.1 KB
Loading

‎articles/aks/resize-node-pool.md

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
---
2+
title: Resize node pools in Azure Kubernetes Service (AKS)
3+
description: Learn how to resize node pools for a cluster in Azure Kubernetes Service (AKS) by cordoning and draining.
4+
services: container-service
5+
ms.topic: how-to
6+
ms.date: 02/24/2022
7+
#Customer intent: As a cluster operator, I want to resize my node pools so that I can run more or larger workloads.
8+
---
9+
10+
# Resize node pools in Azure Kubernetes Service (AKS)
11+
12+
Due to an increasing number of deployments or to run a larger workload, you may want to change the virtual machine scale set plan or resize AKS instances. However, as per [support policies for AKS][aks-support-policies]:
13+
14+
> AKS agent nodes appear in the Azure portal as regular Azure IaaS resources. But these virtual machines are deployed into a custom Azure resource group (usually prefixed with MC_*). You cannot do any direct customizations to these nodes using the IaaS APIs or resources. Any custom changes that are not done via the AKS API will not persist through an upgrade, scale, update or reboot.
15+
16+
This lack of persistence also applies to the resize operation, thus, resizing AKS instances in this manner isn't supported. In this how-to guide, you'll learn the recommended method to address this scenario.
17+
18+
> [!IMPORTANT]
19+
> This method is specific to virtual machine scale set-based AKS clusters. When using virtual machine availability sets, you are limited to only one node pool per cluster.
20+
21+
## Example resources
22+
23+
Suppose you want to resize an existing node pool, called `nodepool1`, from SKU size Standard_DS2_v2 to Standard_DS3_v2. To accomplish this task, you'll need to create a new node pool using Standard_DS3_v2, move workloads from `nodepool1` to the new node pool, and remove `nodepool1`. In this example, we'll call this new node pool `mynodepool`.
24+
25+
:::image type="content" source="./media/resize-node-pool/node-pool-ds2.png" alt-text="The Azure portal page for the cluster, navigated to Settings > Node pools. One node pool, named node pool 1, is shown.":::
26+
27+
```bash
28+
kubectl get nodes
29+
30+
NAME STATUS ROLES AGE VERSION
31+
aks-nodepool1-31721111-vmss000000 Ready agent 10d v1.21.9
32+
aks-nodepool1-31721111-vmss000001 Ready agent 10d v1.21.9
33+
aks-nodepool1-31721111-vmss000002 Ready agent 10d v1.21.9
34+
```
35+
36+
```bash
37+
kubectl get pods -o wide -A
38+
39+
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
40+
default sampleapp2-74b4b974ff-676sz 1/1 Running 0 93m 10.244.1.6 aks-nodepool1-31721111-vmss000002 <none> <none>
41+
default sampleapp2-76b6c4c59b-pfgbh 1/1 Running 0 94m 10.244.1.5 aks-nodepool1-31721111-vmss000002 <none> <none>
42+
kube-system azure-ip-masq-agent-4n66k 1/1 Running 0 10d 10.240.0.6 aks-nodepool1-31721111-vmss000002 <none> <none>
43+
kube-system azure-ip-masq-agent-9p4c8 1/1 Running 0 10d 10.240.0.4 aks-nodepool1-31721111-vmss000000 <none> <none>
44+
kube-system azure-ip-masq-agent-nb7mx 1/1 Running 0 10d 10.240.0.5 aks-nodepool1-31721111-vmss000001 <none> <none>
45+
kube-system coredns-845757d86-dtvvs 1/1 Running 0 10d 10.244.0.2 aks-nodepool1-31721111-vmss000000 <none> <none>
46+
kube-system coredns-845757d86-x27pp 1/1 Running 0 10d 10.244.2.3 aks-nodepool1-31721111-vmss000001 <none> <none>
47+
kube-system coredns-autoscaler-5f85dc856b-nfrmh 1/1 Running 0 10d 10.244.2.4 aks-nodepool1-31721111-vmss000001 <none> <none>
48+
kube-system csi-azuredisk-node-9nfzt 3/3 Running 0 10d 10.240.0.4 aks-nodepool1-31721111-vmss000000 <none> <none>
49+
kube-system csi-azuredisk-node-bblsb 3/3 Running 0 10d 10.240.0.5 aks-nodepool1-31721111-vmss000001 <none> <none>
50+
kube-system csi-azuredisk-node-tjhj4 3/3 Running 0 10d 10.240.0.6 aks-nodepool1-31721111-vmss000002 <none> <none>
51+
kube-system csi-azurefile-node-9pcr8 3/3 Running 0 3d10h 10.240.0.6 aks-nodepool1-31721111-vmss000002 <none> <none>
52+
kube-system csi-azurefile-node-bh2pc 3/3 Running 0 3d10h 10.240.0.5 aks-nodepool1-31721111-vmss000001 <none> <none>
53+
kube-system csi-azurefile-node-h75gq 3/3 Running 0 3d10h 10.240.0.4 aks-nodepool1-31721111-vmss000000 <none> <none>
54+
kube-system konnectivity-agent-6cd55c69cf-ngdlb 1/1 Running 0 10d 10.240.0.6 aks-nodepool1-31721111-vmss000002 <none> <none>
55+
kube-system konnectivity-agent-6cd55c69cf-rvvqt 1/1 Running 0 10d 10.240.0.4 aks-nodepool1-31721111-vmss000000 <none> <none>
56+
kube-system kube-proxy-4wzx7 1/1 Running 0 10d 10.240.0.4 aks-nodepool1-31721111-vmss000000 <none> <none>
57+
kube-system kube-proxy-g5tvr 1/1 Running 0 10d 10.240.0.6 aks-nodepool1-31721111-vmss000002 <none> <none>
58+
kube-system kube-proxy-mrv54 1/1 Running 0 10d 10.240.0.5 aks-nodepool1-31721111-vmss000001 <none> <none>
59+
kube-system metrics-server-774f99dbf4-h52hn 1/1 Running 1 3d10h 10.244.1.3 aks-nodepool1-31721111-vmss000002 <none> <none>
60+
```
61+
62+
## Create a new node pool with the desired SKU
63+
64+
Use the [az aks nodepool add][az-aks-nodepool-add] command to create a new node pool called `mynodepool` with three nodes using the `Standard_DS3_v2` VM SKU:
65+
66+
```azurecli-interactive
67+
az aks nodepool add \
68+
--resource-group myResourceGroup \
69+
--cluster-name myAKSCluster \
70+
--name mynodepool \
71+
--node-count 3 \
72+
--node-vm-size Standard_DS3_v2 \
73+
--mode System \
74+
--no-wait
75+
```
76+
77+
> [!NOTE]
78+
> Every AKS cluster must contain at least one system node pool with at least one node. In the below example, we are using a `--mode` of `System`, as the cluster is assumed to have only one node pool, necessitating a `System` node pool to replace it. A node pool's mode can be [updated at any time][update-node-pool-mode].
79+
80+
When resizing, be sure to consider other requirements and configure your node pool accordingly. You may need to modify the above command. For a full list of the configuration options, see the [az aks nodepool add][az-aks-nodepool-add] reference page.
81+
82+
After a few minutes, the new node pool has been created:
83+
84+
:::image type="content" source="./media/resize-node-pool/node-pool-both.png" alt-text="The Azure portal page for the cluster, navigated to Settings > Node pools. Two node pools, named node pool 1 and my node pool, respectively, are shown.":::
85+
86+
```bash
87+
kubectl get nodes
88+
89+
NAME STATUS ROLES AGE VERSION
90+
aks-mynodepool-20823458-vmss000000 Ready agent 23m v1.21.9
91+
aks-mynodepool-20823458-vmss000001 Ready agent 23m v1.21.9
92+
aks-mynodepool-20823458-vmss000002 Ready agent 23m v1.21.9
93+
aks-nodepool1-31721111-vmss000000 Ready agent 10d v1.21.9
94+
aks-nodepool1-31721111-vmss000001 Ready agent 10d v1.21.9
95+
aks-nodepool1-31721111-vmss000002 Ready agent 10d v1.21.9
96+
```
97+
98+
## Cordon the existing nodes
99+
100+
Cordoning marks specified nodes as unschedulable and prevents any more pods from being added to the nodes.
101+
102+
First, obtain the names of the nodes you'd like to cordon with `kubectl get nodes`. Your output should look similar to the following:
103+
104+
```bash
105+
NAME STATUS ROLES AGE VERSION
106+
aks-nodepool1-31721111-vmss000000 Ready agent 7d21h v1.21.9
107+
aks-nodepool1-31721111-vmss000001 Ready agent 7d21h v1.21.9
108+
aks-nodepool1-31721111-vmss000002 Ready agent 7d21h v1.21.9
109+
```
110+
111+
Next, using `kubectl cordon <node-names>`, specify the desired nodes in a space-separated list:
112+
113+
```bash
114+
kubectl cordon aks-nodepool1-31721111-vmss000000 aks-nodepool1-31721111-vmss000001 aks-nodepool1-31721111-vmss000002
115+
```
116+
117+
```bash
118+
node/aks-nodepool1-31721111-vmss000000 cordoned
119+
node/aks-nodepool1-31721111-vmss000001 cordoned
120+
node/aks-nodepool1-31721111-vmss000002 cordoned
121+
```
122+
123+
## Drain the existing nodes
124+
125+
> [!IMPORTANT]
126+
> To successfully drain nodes and evict running pods, ensure that any PodDisruptionBudgets (PDBs) allow for at least 1 pod replica to be moved at a time, otherwise the drain/evict operation will fail. To check this, you can run `kubectl get pdb -A` and make sure `ALLOWED DISRUPTIONS` is at least 1 or higher.
127+
128+
Draining nodes will cause pods running on them to be evicted and recreated on the other, schedulable nodes.
129+
130+
To drain nodes, use `kubectl drain <node-names> --ignore-daemonsets --delete-emptydir-data`, again using a space-separated list of node names:
131+
132+
> [!IMPORTANT]
133+
> Using `--delete-emptydir-data` is required to evict the AKS-created `coredns` and `metrics-server` pods. If this flag isn't used, an error is expected. For more information, see the [documentation on emptydir][empty-dir].
134+
135+
```bash
136+
kubectl drain aks-nodepool1-31721111-vmss000000 aks-nodepool1-31721111-vmss000001 aks-nodepool1-31721111-vmss000002 --ignore-daemonsets --delete-emptydir-data
137+
```
138+
139+
After the drain operation finishes, all pods other than those controlled by daemon sets are running on the new node pool:
140+
141+
```bash
142+
kubectl get pods -o wide -A
143+
144+
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
145+
default sampleapp2-74b4b974ff-676sz 1/1 Running 0 15m 10.244.4.5 aks-mynodepool-20823458-vmss000002 <none> <none>
146+
default sampleapp2-76b6c4c59b-rhmzq 1/1 Running 0 16m 10.244.4.3 aks-mynodepool-20823458-vmss000002 <none> <none>
147+
kube-system azure-ip-masq-agent-4n66k 1/1 Running 0 10d 10.240.0.6 aks-nodepool1-31721111-vmss000002 <none> <none>
148+
kube-system azure-ip-masq-agent-9p4c8 1/1 Running 0 10d 10.240.0.4 aks-nodepool1-31721111-vmss000000 <none> <none>
149+
kube-system azure-ip-masq-agent-nb7mx 1/1 Running 0 10d 10.240.0.5 aks-nodepool1-31721111-vmss000001 <none> <none>
150+
kube-system azure-ip-masq-agent-sxn96 1/1 Running 0 49m 10.240.0.9 aks-mynodepool-20823458-vmss000002 <none> <none>
151+
kube-system azure-ip-masq-agent-tsq98 1/1 Running 0 49m 10.240.0.8 aks-mynodepool-20823458-vmss000001 <none> <none>
152+
kube-system azure-ip-masq-agent-xzrdl 1/1 Running 0 49m 10.240.0.7 aks-mynodepool-20823458-vmss000000 <none> <none>
153+
kube-system coredns-845757d86-d2pkc 1/1 Running 0 17m 10.244.3.2 aks-mynodepool-20823458-vmss000000 <none> <none>
154+
kube-system coredns-845757d86-f8g9s 1/1 Running 0 17m 10.244.5.2 aks-mynodepool-20823458-vmss000001 <none> <none>
155+
kube-system coredns-autoscaler-5f85dc856b-f8xh2 1/1 Running 0 17m 10.244.4.2 aks-mynodepool-20823458-vmss000002 <none> <none>
156+
kube-system csi-azuredisk-node-7md2w 3/3 Running 0 49m 10.240.0.7 aks-mynodepool-20823458-vmss000000 <none> <none>
157+
kube-system csi-azuredisk-node-9nfzt 3/3 Running 0 10d 10.240.0.4 aks-nodepool1-31721111-vmss000000 <none> <none>
158+
kube-system csi-azuredisk-node-bblsb 3/3 Running 0 10d 10.240.0.5 aks-nodepool1-31721111-vmss000001 <none> <none>
159+
kube-system csi-azuredisk-node-lcmtz 3/3 Running 0 49m 10.240.0.9 aks-mynodepool-20823458-vmss000002 <none> <none>
160+
kube-system csi-azuredisk-node-mmncr 3/3 Running 0 49m 10.240.0.8 aks-mynodepool-20823458-vmss000001 <none> <none>
161+
kube-system csi-azuredisk-node-tjhj4 3/3 Running 0 10d 10.240.0.6 aks-nodepool1-31721111-vmss000002 <none> <none>
162+
kube-system csi-azurefile-node-29w6z 3/3 Running 0 49m 10.240.0.9 aks-mynodepool-20823458-vmss000002 <none> <none>
163+
kube-system csi-azurefile-node-4nrx7 3/3 Running 0 49m 10.240.0.7 aks-mynodepool-20823458-vmss000000 <none> <none>
164+
kube-system csi-azurefile-node-9pcr8 3/3 Running 0 3d11h 10.240.0.6 aks-nodepool1-31721111-vmss000002 <none> <none>
165+
kube-system csi-azurefile-node-bh2pc 3/3 Running 0 3d11h 10.240.0.5 aks-nodepool1-31721111-vmss000001 <none> <none>
166+
kube-system csi-azurefile-node-gqqnv 3/3 Running 0 49m 10.240.0.8 aks-mynodepool-20823458-vmss000001 <none> <none>
167+
kube-system csi-azurefile-node-h75gq 3/3 Running 0 3d11h 10.240.0.4 aks-nodepool1-31721111-vmss000000 <none> <none>
168+
kube-system konnectivity-agent-6cd55c69cf-2bbp5 1/1 Running 0 17m 10.240.0.7 aks-mynodepool-20823458-vmss000000 <none> <none>
169+
kube-system konnectivity-agent-6cd55c69cf-7xzxj 1/1 Running 0 16m 10.240.0.8 aks-mynodepool-20823458-vmss000001 <none> <none>
170+
kube-system kube-proxy-4wzx7 1/1 Running 0 10d 10.240.0.4 aks-nodepool1-31721111-vmss000000 <none> <none>
171+
kube-system kube-proxy-7h8r5 1/1 Running 0 49m 10.240.0.7 aks-mynodepool-20823458-vmss000000 <none> <none>
172+
kube-system kube-proxy-g5tvr 1/1 Running 0 10d 10.240.0.6 aks-nodepool1-31721111-vmss000002 <none> <none>
173+
kube-system kube-proxy-mrv54 1/1 Running 0 10d 10.240.0.5 aks-nodepool1-31721111-vmss000001 <none> <none>
174+
kube-system kube-proxy-nqmnj 1/1 Running 0 49m 10.240.0.9 aks-mynodepool-20823458-vmss000002 <none> <none>
175+
kube-system kube-proxy-zn77s 1/1 Running 0 49m 10.240.0.8 aks-mynodepool-20823458-vmss000001 <none> <none>
176+
kube-system metrics-server-774f99dbf4-2x6x8 1/1 Running 0 16m 10.244.4.4 aks-mynodepool-20823458-vmss000002 <none> <none>
177+
```
178+
179+
### Troubleshooting
180+
181+
You may see an error like the following:
182+
> Error when evicting pods/[podname] -n [namespace] (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
183+
184+
By default, your cluster has AKS_managed pod disruption budgets (such as `coredns-pdb` or `konnectivity-agent`) with a `MinAvailable` of 1. If, for example, there are two `coredns` pods running, while one of them is getting recreated and is unavailable, the other is unable to be affected due to the pod disruption budget. This resolves itself after the initial `coredns` pod is scheduled and running, allowing the second pod to be properly evicted and recreated.
185+
186+
> [!TIP]
187+
> Consider draining nodes one-by-one for a smoother eviction experience and to avoid throttling. For more information, see:
188+
> * [Plan for availability using a pod disruption budget][pod-disruption-budget]
189+
> * [Specifying a Disruption Budget for your Application][specify-disruption-budget]
190+
> * [Disruptions][disruptions]
191+
192+
## Remove the existing node pool
193+
194+
To delete the existing node pool, use the Azure portal or the [az aks delete][az-aks-delete] command:
195+
196+
```bash
197+
kubectl delete nodepool /
198+
--resource-group myResourceGroup /
199+
--cluster-name myAKSCluster /
200+
--name nodepool1
201+
```
202+
203+
After completion, the final result is the AKS cluster having a single, new node pool with the new, desired SKU size and all the applications and pods properly running:
204+
205+
:::image type="content" source="./media/resize-node-pool/node-pool-ds3.png" alt-text="The Azure portal page for the cluster, navigated to Settings > Node pools. One node pool, named my node pool, is shown.":::
206+
207+
```bash
208+
kubectl get nodes
209+
210+
NAME STATUS ROLES AGE VERSION
211+
aks-mynodepool-20823458-vmss000000 Ready agent 63m v1.21.9
212+
aks-mynodepool-20823458-vmss000001 Ready agent 63m v1.21.9
213+
aks-mynodepool-20823458-vmss000002 Ready agent 63m v1.21.9
214+
```
215+
216+
## Next steps
217+
218+
After resizing a node pool by cordoning and draining, learn more about [using multiple node pools][use-multiple-node-pools].
219+
220+
<!-- LINKS -->
221+
[az-aks-nodepool-add]: /cli/azure/nodepool#az_aks_nodepool_add
222+
[az-aks-delete]: /cli/azure/nodepool#az_aks_delete
223+
[aks-support-policies]: support-policies.md#user-customization-of-agent-nodes
224+
[update-node-pool-mode]: use-system-pools.md#update-existing-cluster-system-and-user-node-pools
225+
[pod-disruption-budget]: operator-best-practices-scheduler.md#plan-for-availability-using-pod-disruption-budgets
226+
[empty-dir]: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
227+
[specify-disruption-budget]: https://kubernetes.io/docs/tasks/run-application/configure-pdb/
228+
[disruptions]: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
229+
[use-multiple-node-pools]: use-multiple-node-pools.md

0 commit comments

Comments
 (0)