Skip to content

Commit 4134a06

Browse files
MGoedtelMGoedtel
committed
Metrics Server VPA article
1 parent f5ca4d1 commit 4134a06

File tree

3 files changed

+210
-48
lines changed

3 files changed

+210
-48
lines changed

‎articles/aks/TOC.yml

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -199,8 +199,10 @@
199199
href: aks-diagnostics.md
200200
- name: Integrate ACR with an AKS cluster
201201
href: cluster-container-registry-integration.md
202-
- name: Use Vertical Pod Autoscaler (preview)
202+
- name: Use Vertical Pod Autoscaler
203203
href: vertical-pod-autoscaler.md
204+
- name: Metrics Server
205+
href: use-metrics-server-vertical-pod-autoscaler.md
204206
- name: Scale an AKS cluster
205207
href: scale-cluster.md
206208
- name: Stop/Deallocate nodes with Scale-down Mode
@@ -349,15 +351,15 @@
349351
items:
350352
- name: Create an OIDC Issuer for your cluster
351353
href: use-oidc-issuer.md
352-
- name: Workload identity (preview)
354+
- name: Workload identity
353355
items:
354356
- name: Overview
355357
href: workload-identity-overview.md
356358
- name: Deploy and configure cluster
357359
href: workload-identity-deploy-cluster.md
358360
- name: Modernize your app with workload identity
359361
href: workload-identity-migrate-from-pod-identity.md
360-
- name: Use Azure AD pod identity (preview)
362+
- name: Use Azure AD pod identity
361363
href: use-azure-ad-pod-identity.md
362364
- name: Use Pod Sandboxing
363365
href: use-pod-sandboxing.md

‎articles/aks/use-metrics-server-vertical-pod-autoscaling.md

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
---
2+
title: Metrics server VPA throttleing in Azure Kubernetes Service (AKS)
3+
description: Learn how to vertically autoscale your pod on an Azure Kubernetes Service (AKS) cluster.
4+
ms.topic: article
5+
ms.custom: devx-track-azurecli
6+
ms.date: 03/17/2023
7+
---
8+
9+
# Metrics server VPA throttling in Azure Kubernetes Service (AKS)
10+
11+
[Metrics Server][metrics-server-overview] is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines. With Azure Kubernetes Service (AKS), vertical pod autoscaling is enabled for the Metrics Server. The Metrics Server is commonly used by other Kubernetes add ons, such as the [Horizontal Pod Autoscaler][horizontal-pod-autoscaler].
12+
13+
Vertical Pod Autoscaler (VPA) enables you to adjust the resource limit when the Metrics Server is experiencing consistent CPU and memory resource constraints.
14+
15+
## Before you begin
16+
17+
AKS cluster is running Kubernetes version 1.24 and higher.
18+
19+
## Metrics server crashloopbackoff or throttling
20+
21+
If the Metrics Server throttling rate is high and the memory usage of its two pods are unbalanced, this indicates the Metrics Server requires more resources than the default values specified.
22+
23+
To update the coefficient values, create a ConfigMap in the overlay *kube-system* namespace to override the values in the Metrics Server specification. Perform the following steps to update the metrics server.
24+
25+
1. Create a ConfigMap file named *metrics-server-config.yaml* and copy in the following manifest.
26+
27+
```yml
28+
apiVersion: v1
29+
kind: ConfigMap
30+
metadata:
31+
name: metrics-server-config
32+
namespace: kube-system
33+
labels:
34+
kubernetes.io/cluster-service: "true"
35+
addonmanager.kubernetes.io/mode: EnsureExists
36+
data:
37+
NannyConfiguration: |-
38+
apiVersion: nannyconfig/v1alpha1
39+
kind: NannyConfiguration
40+
baseCPU: 100m
41+
cpuPerNode: 1m
42+
baseMemory: 100Mi
43+
memoryPerNode: 8Mi
44+
```
45+
46+
In this ConfigMap example, it changes the resource limit and request to the following:
47+
48+
* cpu: (100+1n) millicore
49+
* memory: (100+8n) mebibyte
50+
51+
Where *n* is the number of nodes.
52+
53+
2. Create the ConfigMap using the [kubectl apply][kubectl-apply] command and specify the name of your YAML manifest:
54+
55+
```bash
56+
kubectl apply -f metrics-server-config.yaml
57+
```
58+
59+
3. Restart the Metrics Server pods. There are two Metrics server pods, and the following command deletes all of them.
60+
61+
```bash
62+
kubectl -n kube-system delete po metrics-server-pod-name
63+
```
64+
65+
4. To verify the updated resources took affect, run the following command to review the Metrics Server VPA log.
66+
67+
```bash
68+
kubectl -n kube-system logs metrics-server-pod-name -c metrics-server-vpa
69+
```
70+
71+
The following example output resembles the results showing the updated throttling settings were applied.
72+
73+
```output
74+
ERROR: logging before flag.Parse: I0315 23:12:33.956112 1 pod_nanny.go:68] Invoked by [/pod_nanny --config-dir=/etc/config --cpu=44m --extra-cpu=0.5m --memory=51Mi --extra-memory=4Mi --poll-period=180000 --threshold=5 --deployment=metrics-server --container=metrics-server]
75+
ERROR: logging before flag.Parse: I0315 23:12:33.956159 1 pod_nanny.go:69] Version: 1.8.14
76+
ERROR: logging before flag.Parse: I0315 23:12:33.956171 1 pod_nanny.go:85] Watching namespace: kube-system, pod: metrics-server-545d8b77b7-5nqq9, container: metrics-server.
77+
ERROR: logging before flag.Parse: I0315 23:12:33.956175 1 pod_nanny.go:86] storage: MISSING, extra_storage: 0Gi
78+
ERROR: logging before flag.Parse: I0315 23:12:33.957441 1 pod_nanny.go:116] cpu: 100m, extra_cpu: 1m, memory: 100Mi, extra_memory: 8Mi
79+
ERROR: logging before flag.Parse: I0315 23:12:33.957456 1 pod_nanny.go:145] Resources: [{Base:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI} ExtraPerNode:{i:{value:0 scale:-3} d:{Dec:<nil>} s: Format:DecimalSI} Name:cpu} {Base:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI} ExtraPerNode:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Name:memory
80+
```
81+
82+
Be cautious of the *baseCPU*, *cpuPerNode*, *baseMemory*, and the *memoryPerNode* as the ConfigMap won't be validated by AKS. As a recommended practice, increase the value gradually to avoid unnecessary resource consumption. Proactively monitor resource usage when updating or creating the ConfigMap. A large number of resource requests could negatively impact the node.
83+
84+
## Manually configure Metrics Server resource usage
85+
86+
The Metrics Server VPA adjusts resource usage by the number of nodes. If the cluster scale up or down very often, the Metrics Server might restart frequently. In this case, you can bypass VPA and manually control its resource usage. This method to configure VPA is not to be performed in addition to the steps described in the previous section.
87+
88+
If you would like to bypass VPA for Metrics Server and manually control its resource usage, perform the following steps.
89+
90+
1. Create a ConfigMap file named *metrics-server-config.yaml* and copy in the following manifest.
91+
92+
```yml
93+
apiVersion: v1
94+
kind: ConfigMap
95+
metadata:
96+
name: metrics-server-config
97+
namespace: kube-system
98+
labels:
99+
kubernetes.io/cluster-service: "true"
100+
addonmanager.kubernetes.io/mode: EnsureExists
101+
data:
102+
NannyConfiguration: |-
103+
apiVersion: nannyconfig/v1alpha1
104+
kind: NannyConfiguration
105+
baseCPU: 100m
106+
cpuPerNode: 0m
107+
baseMemory: 100Mi
108+
memoryPerNode: 0Mi
109+
```
110+
111+
In this ConfigMap example, it changes the resource limit and request to the following:
112+
113+
* cpu: 100 millicore
114+
* memory: 100 mebibyte
115+
116+
So changing the number of nodes will not trigger the autoscaling.
117+
118+
2. Create the ConfigMap using the [kubectl apply][kubectl-apply] command and specify the name of your YAML manifest:
119+
120+
```yml
121+
kubectl apply -f metrics-server-config.yaml
122+
```
123+
124+
3. Restart the Metrics Server pods. There are two Metrics server pods, and the following command deletes all of them.
125+
126+
```bash
127+
kubectl -n kube-system delete po metrics-server-pod-name
128+
```
129+
130+
4. To verify the updated resources took affect, run the following command to review the Metrics Server VPA log.
131+
132+
```bash
133+
kubectl -n kube-system logs metrics-server-pod-name -c metrics-server-vpa
134+
```
135+
136+
The following example output resembles the results showing the updated throttling settings were applied.
137+
138+
```output
139+
ERROR: logging before flag.Parse: I0315 23:12:33.956112 1 pod_nanny.go:68] Invoked by [/pod_nanny --config-dir=/etc/config --cpu=44m
140+
--extra-cpu=0.5m --memory=51Mi --extra-memory=4Mi --poll-period=180000 --threshold=5 --deployment=metrics-server --container=metrics-server]
141+
ERROR: logging before flag.Parse: I0315 23:12:33.956159 1 pod_nanny.go:69] Version: 1.8.14
142+
ERROR: logging before flag.Parse: I0315 23:12:33.956171 1 pod_nanny.go:85] Watching namespace: kube-system, pod: metrics-server-545d8b77b7-5nqq9, container: metrics-server.
143+
ERROR: logging before flag.Parse: I0315 23:12:33.956175 1 pod_nanny.go:86] storage: MISSING, extra_storage: 0Gi
144+
ERROR: logging before flag.Parse: I0315 23:12:33.957441 1 pod_nanny.go:116] cpu: 100m, extra_cpu: 0m, memory: 100Mi, extra_memory: 0Mi
145+
ERROR: logging before flag.Parse: I0315 23:12:33.957456 1 pod_nanny.go:145] Resources: [{Base:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI} ExtraPerNode:{i:{value:0 scale:-3} d:{Dec:<nil>} s: Format:DecimalSI} Name:cpu} {Base:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}
146+
ExtraPerNode:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Name:memory}]
147+
```
148+
149+
## Troubleshooting
150+
151+
1. If you use the following configmap, the Metrics Server VPA customization won't work. You need add a unit for `baseCPU`.
152+
153+
```yml
154+
apiVersion: v1
155+
kind: ConfigMap
156+
metadata:
157+
name: metrics-server-config
158+
namespace: kube-system
159+
labels:
160+
kubernetes.io/cluster-service: "true"
161+
addonmanager.kubernetes.io/mode: EnsureExists
162+
data:
163+
NannyConfiguration: |-
164+
apiVersion: nannyconfig/v1alpha1
165+
kind: NannyConfiguration
166+
baseCPU: 100
167+
cpuPerNode: 1m
168+
baseMemory: 100Mi
169+
memoryPerNode: 8Mi
170+
```
171+
172+
The following example output resembles the results showing the updated throttling settings were not applied.
173+
174+
```output
175+
ERROR: logging before flag.Parse: I0316 23:32:08.383389 1 pod_nanny.go:68] Invoked by [/pod_nanny --config-dir=/etc/config --cpu=44m
176+
--extra-cpu=0.5m --memory=51Mi --extra-memory=4Mi --poll-period=180000 --threshold=5 --deployment=metrics-server --container=metrics-server]
177+
ERROR: logging before flag.Parse: I0316 23:32:08.383430 1 pod_nanny.go:69] Version: 1.8.14
178+
ERROR: logging before flag.Parse: I0316 23:32:08.383441 1 pod_nanny.go:85] Watching namespace: kube-system, pod: metrics-server-7d78876589-hcrff, container: metrics-server.
179+
ERROR: logging before flag.Parse: I0316 23:32:08.383446 1 pod_nanny.go:86] storage: MISSING, extra_storage: 0Gi
180+
ERROR: logging before flag.Parse: I0316 23:32:08.384554 1 pod_nanny.go:192] Unable to decode Nanny Configuration from config map, using default parameters
181+
ERROR: logging before flag.Parse: I0316 23:32:08.384565 1 pod_nanny.go:116] cpu: 44m, extra_cpu: 0.5m, memory: 51Mi, extra_memory: 4Mi
182+
ERROR: logging before flag.Parse: I0316 23:32:08.384589 1 pod_nanny.go:145] Resources: [{Base:{i:{value:44 scale:-3} d:{Dec:<nil>} s:44m Format:DecimalSI} ExtraPerNode:{i:{value:5 scale:-4} d:{Dec:<nil>} s: Format:DecimalSI} Name:cpu} {Base:{i:{value:53477376 scale:0} d:{Dec:<nil>} s:51Mi Format:BinarySI} ExtraPerNode:{i:{value:4194304 scale:0}
183+
d:{Dec:<nil>} s:4Mi Format:BinarySI} Name:memory}]
184+
```
185+
186+
2. For Kubernetes version 1.23 and higher clusters, Metrics Server has a *PodDisruptionBudget*. It ensures the number of available Metrics Server pods is at least one. If you get something like this after running `kubectl -n kube-system get po`, it's possible that the customized resource usage is small. Increase the coefficient values to resolve it.
187+
188+
```output
189+
metrics-server-679b886d4-pxwdf 1/2 CrashLoopBackOff 6 (36s ago) 6m33s
190+
metrics-server-679b886d4-svxxx 1/2 CrashLoopBackOff 6 (54s ago) 6m33s
191+
metrics-server-7d78876589-hcrff 2/2 Running 0 37m
192+
```
193+
194+
## Next steps
195+
196+
Metrics Server is a component in the core metrics pipeline. For more information see [Metrics Server API design][metrics-server-api-design].
197+
198+
<!-- EXTERNAL LINKS -->
199+
[kubectl-apply]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#apply
200+
[metrics-server-overview]: https://kubernetes-sigs.github.io/metrics-server/
201+
[metrics-server-api-design]: https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/resource-metrics-api.md
202+
203+
<!--- INTERNAL LINKS --->
204+
[horizontal-pod-autoscaler]: concepts-scale.md#horizontal-pod-autoscaler

‎articles/aks/vertical-pod-autoscaler.md

Lines changed: 1 addition & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: Vertical Pod Autoscaling (preview) in Azure Kubernetes Service (AKS)
33
description: Learn how to vertically autoscale your pod on an Azure Kubernetes Service (AKS) cluster.
44
ms.topic: article
55
ms.custom: devx-track-azurecli
6-
ms.date: 01/12/2023
6+
ms.date: 03/17/2023
77
---
88

99
# Vertical Pod Autoscaling (preview) in Azure Kubernetes Service (AKS)
@@ -394,50 +394,6 @@ Vertical Pod autoscaling uses the `VerticalPodAutoscaler` object to automaticall
394394

395395
The Vertical Pod Autoscaler uses the `lowerBound` and `upperBound` attributes to decide whether to delete a Pod and replace it with a new Pod. If a Pod has requests less than the lower bound or greater than the upper bound, the Vertical Pod Autoscaler deletes the Pod and replaces it with a Pod that meets the target attribute.
396396

397-
## Metrics server VPA throttling
398-
399-
With AKS clusters version 1.24 and higher, vertical pod autoscaling is enabled for the metrics server. VPA enables you to adjust the resource limit when the metrics server is experiencing consistent CPU and memory resource constraints.
400-
401-
If the metrics server throttling rate is high and the memory usage of its two pods are unbalanced, this indicates the metrics server requires more resources than the default values specified.
402-
403-
To update the coefficient values, create a ConfigMap in the overlay *kube-system* namespace to override the values in the metrics server specification. Perform the following steps to update the metrics server.
404-
405-
1. Create a ConfigMap file named *metrics-server-config.yaml* and copy in the following manifest.
406-
407-
```yml
408-
apiVersion: v1
409-
kind: ConfigMap
410-
metadata:
411-
name: metrics-server-config
412-
namespace: kube-system
413-
labels:
414-
kubernetes.io/cluster-service: "true"
415-
addonmanager.kubernetes.io/mode: EnsureExists
416-
data:
417-
NannyConfiguration: |-
418-
apiVersion: nannyconfig/v1alpha1
419-
kind: NannyConfiguration
420-
baseCPU: 100m
421-
cpuPerNode: 1m
422-
baseMemory: 100Mi
423-
memoryPerNode: 8Mi
424-
```
425-
426-
In this ConfigMap example, it changes the resource limit and request to the following:
427-
428-
* cpu: (100+1n) millicore
429-
* memory: (100+8n) mebibyte
430-
431-
Where *n* is the number of nodes.
432-
433-
2. Create the ConfigMap using the [kubectl apply][kubectl-apply] command and specify the name of your YAML manifest:
434-
435-
```bash
436-
kubectl apply -f metrics-server-config.yaml
437-
```
438-
439-
Be cautious of the *baseCPU*, *cpuPerNode*, *baseMemory*, and the *memoryPerNode* as the ConfigMap won't be validated by AKS. As a recommended practice, increase the value gradually to avoid unnecessary resource consumption. Proactively monitor resource usage when updating or creating the ConfigMap. A large number of resource requests could negatively impact the node.
440-
441397
## Next steps
442398

443399
This article showed you how to automatically scale resource utilization, such as CPU and memory, of cluster nodes to match application requirements. You can also use the horizontal pod autoscaler to automatically adjust the number of pods that run your application. For steps on using the horizontal pod autoscaler, see [Scale applications in AKS][scale-applications-in-aks].

0 commit comments

Comments
 (0)