Skip to content

Commit 3cd8c0b

Browse files
umagnusumagnus
committed
add vpa to blob csi
1 parent b0ce2e9 commit 3cd8c0b

14 files changed

+1805
-1
lines changed

‎deploy/example/vpa/README.md‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Vertical pod autoscaler to Azure blob CSI controller pods
2+
## Prerequisites
3+
### Step 1: Install Vertical Pod Autoscaler (VPA)
4+
- **Recommend**: You can use the script located at `blob-csi-driver/deploy/example/vpa/install-vpa.sh` to install the VPA. If you use this method, you can skip **Step 2**.
5+
- You can also refer to the github repo [vertical-pod-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/README.md) for installation instructions.
6+
7+
8+
### Step 2: Adjust Admission Controller Webhooks in AKS
9+
In AKS, you need to add label `admissions.enforcer/disabled: true` to admission controller webhooks to impact kube-system AKS namespaces, refer to [Can admission controller webhooks impact kube-system and internal AKS namespaces?](https://learn.microsoft.com/en-us/azure/aks/faq#can-admission-controller-webhooks-impact-kube-system-and-internal-aks-namespaces-). This can be done by using the `--webhook-labels` flag in `vpa-admission-controller`, refer to [Running the admission-controller](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/components.md#:~:text=You%20can%20specify%20a%20comma%20separated%20list%20to%20set%20webhook%20labels%20with%20%2D%2Dwebhook%2Dlabels%2C%20example%20format%3A%20key1%3Avalue1%2Ckey2%3Avalue2.)
10+
11+
If you do not use the recommended script to install VPA, you should manually add the `--webhook-labels=admissions.enforcer/disabled:true` flag in the vpa-admission-controller deployment.
12+
13+
> edit vpa-admission-controller deployment
14+
```
15+
k edit deploy vpa-admission-controller -n kube-system
16+
```
17+
> add `- --webhook-labels=admissions.enforcer/disabled:true` in containers args
18+
```
19+
template:
20+
metadata:
21+
creationTimestamp: null
22+
labels:
23+
app: vpa-admission-controller
24+
spec:
25+
containers:
26+
- args:
27+
- --v=4
28+
- --stderrthreshold=info
29+
- --reload-cert
30+
- --webhook-labels=admissions.enforcer/disabled:true # add webhook-labels flags
31+
```
32+
33+
> check vpa-admission-controller pod running
34+
```
35+
k get po -n kube-system | grep vpa-admission-controller
36+
vpa-admission-controller-7fcb5c6b86-s69p8 1/1 Running 0 13m
37+
```
38+
39+
## Create a VPA corresponding to CSI controller deployment
40+
> create a VPA for CSI controller
41+
```console
42+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/vpa/vertical-pod-autoscaler.yaml
43+
```
44+
> check the VPA config and current recommended resource requests, have 2 evictedPod events for example
45+
```
46+
kubectl get vpa -n kube-system
47+
NAME MODE CPU MEM PROVIDED AGE
48+
csi-blob-controller Auto 15m 43690666 True 8s
49+
50+
kubectl describe vpa csi-blob-controller -n kube-system
51+
Name: csi-blob-controller
52+
Namespace: kube-system
53+
Labels: <none>
54+
Annotations: <none>
55+
API Version: autoscaling.k8s.io/v1
56+
Kind: VerticalPodAutoscaler
57+
Metadata:
58+
Creation Timestamp: 2025-03-25T06:45:04Z
59+
Generation: 1
60+
Resource Version: 62488
61+
UID: 1675169f-6fd9-4c7c-b33e-fffe4f6fc57c
62+
Spec:
63+
Resource Policy:
64+
Container Policies:
65+
Container Name: *
66+
Controlled Resources:
67+
memory
68+
Max Allowed:
69+
Memory: 10Gi
70+
Target Ref:
71+
API Version: apps/v1
72+
Kind: Deployment
73+
Name: csi-blob-controller
74+
Update Policy:
75+
Update Mode: Auto
76+
Status:
77+
Conditions:
78+
Last Transition Time: 2025-03-25T06:45:15Z
79+
Status: True
80+
Type: RecommendationProvided
81+
Recommendation:
82+
Container Recommendations:
83+
Container Name: blob
84+
Lower Bound:
85+
Memory: 43690666
86+
Target:
87+
Memory: 43690666
88+
Uncapped Target:
89+
Memory: 43690666
90+
Upper Bound:
91+
Memory: 10Gi
92+
Container Name: csi-attacher
93+
Lower Bound:
94+
Memory: 43690666
95+
Target:
96+
Memory: 43690666
97+
Uncapped Target:
98+
Memory: 43690666
99+
Upper Bound:
100+
Memory: 10Gi
101+
Container Name: csi-provisioner
102+
Lower Bound:
103+
Memory: 43690666
104+
Target:
105+
Memory: 43690666
106+
Uncapped Target:
107+
Memory: 43690666
108+
Upper Bound:
109+
Memory: 10Gi
110+
Container Name: csi-resizer
111+
Lower Bound:
112+
Memory: 43690666
113+
Target:
114+
Memory: 43690666
115+
Uncapped Target:
116+
Memory: 43690666
117+
Upper Bound:
118+
Memory: 10Gi
119+
Container Name: csi-snapshotter
120+
Lower Bound:
121+
Memory: 43690666
122+
Target:
123+
Memory: 43690666
124+
Uncapped Target:
125+
Memory: 43690666
126+
Upper Bound:
127+
Memory: 10Gi
128+
Container Name: liveness-probe
129+
Lower Bound:
130+
Memory: 43690666
131+
Target:
132+
Memory: 43690666
133+
Uncapped Target:
134+
Memory: 43690666
135+
Upper Bound:
136+
Memory: 10Gi
137+
Events:
138+
Type Reason Age From Message
139+
---- ------ ---- ---- -------
140+
Normal EvictedPod 101s vpa-updater VPA Updater evicted Pod csi-blob-controller-6658fb5fdc-d5mtr to apply resource recommendation.
141+
Normal EvictedPod 41s vpa-updater VPA Updater evicted Pod csi-blob-controller-6658fb5fdc-hpdfk to apply resource recommendation.
142+
```

‎deploy/example/vpa/install-components/admission-controller-deployment.yaml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
---
2+
apiVersion: apps/v1
3+
kind: Deployment
4+
metadata:
5+
name: vpa-admission-controller
6+
namespace: kube-system
7+
spec:
8+
replicas: 1
9+
selector:
10+
matchLabels:
11+
app: vpa-admission-controller
12+
template:
13+
metadata:
14+
labels:
15+
app: vpa-admission-controller
16+
spec:
17+
serviceAccountName: vpa-admission-controller
18+
securityContext:
19+
runAsNonRoot: true
20+
runAsUser: 65534 # nobody
21+
containers:
22+
- name: admission-controller
23+
image: registry.k8s.io/autoscaling/vpa-admission-controller:1.3.0
24+
imagePullPolicy: IfNotPresent
25+
env:
26+
- name: NAMESPACE
27+
valueFrom:
28+
fieldRef:
29+
fieldPath: metadata.namespace
30+
args: ["--v=4", "--stderrthreshold=info", "--reload-cert", "--webhook-labels=admissions.enforcer/disabled:true"]
31+
volumeMounts:
32+
- name: tls-certs
33+
mountPath: "/etc/tls-certs"
34+
readOnly: true
35+
resources:
36+
limits:
37+
cpu: 200m
38+
memory: 500Mi
39+
requests:
40+
cpu: 50m
41+
memory: 200Mi
42+
ports:
43+
- containerPort: 8000
44+
- name: prometheus
45+
containerPort: 8944
46+
volumes:
47+
- name: tls-certs
48+
secret:
49+
secretName: vpa-tls-certs
50+
---
51+
apiVersion: v1
52+
kind: Service
53+
metadata:
54+
name: vpa-webhook
55+
namespace: kube-system
56+
spec:
57+
ports:
58+
- port: 443
59+
targetPort: 8000
60+
selector:
61+
app: vpa-admission-controller

‎deploy/example/vpa/install-components/delete-webhook.sh‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/bash
2+
3+
# Copyright 2018 The Kubernetes Authors.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# Unregisters the admission controller webhook.
18+
set -e
19+
20+
echo "Unregistering VPA admission controller webhook"
21+
22+
kubectl delete -n kube-system mutatingwebhookconfiguration.v1.admissionregistration.k8s.io vpa-webhook-config --ignore-not-found

‎deploy/example/vpa/install-components/gencerts.sh‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
#!/bin/bash
2+
3+
# Copyright 2018 The Kubernetes Authors.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# Generates the a CA cert, a server key, and a server cert signed by the CA.
18+
# reference:
19+
# https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/webhook/gencerts.sh
20+
set -o errexit
21+
set -o nounset
22+
set -o pipefail
23+
24+
CN_BASE="vpa_webhook"
25+
TMP_DIR="/tmp/vpa-certs"
26+
27+
echo "Generating certs for the VPA Admission Controller in ${TMP_DIR}."
28+
mkdir -p ${TMP_DIR}
29+
cat > ${TMP_DIR}/server.conf << EOF
30+
[req]
31+
req_extensions = v3_req
32+
distinguished_name = req_distinguished_name
33+
[req_distinguished_name]
34+
[ v3_req ]
35+
basicConstraints = CA:FALSE
36+
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
37+
extendedKeyUsage = clientAuth, serverAuth
38+
subjectAltName = DNS:vpa-webhook.kube-system.svc
39+
EOF
40+
41+
# Create a certificate authority
42+
openssl genrsa -out ${TMP_DIR}/caKey.pem 2048
43+
set +o errexit
44+
openssl req -x509 -new -nodes -key ${TMP_DIR}/caKey.pem -days 100000 -out ${TMP_DIR}/caCert.pem -subj "/CN=${CN_BASE}_ca" -addext "subjectAltName = DNS:${CN_BASE}_ca"
45+
if [[ $? -ne 0 ]]; then
46+
echo "ERROR: Failed to create CA certificate for self-signing. If the error is \"unknown option -addext\", update your openssl version or deploy VPA from the vpa-release-0.8 branch."
47+
exit 1
48+
fi
49+
set -o errexit
50+
51+
# Create a server certificate
52+
openssl genrsa -out ${TMP_DIR}/serverKey.pem 2048
53+
# Note the CN is the DNS name of the service of the webhook.
54+
openssl req -new -key ${TMP_DIR}/serverKey.pem -out ${TMP_DIR}/server.csr -subj "/CN=vpa-webhook.kube-system.svc" -config ${TMP_DIR}/server.conf
55+
openssl x509 -req -in ${TMP_DIR}/server.csr -CA ${TMP_DIR}/caCert.pem -CAkey ${TMP_DIR}/caKey.pem -CAcreateserial -out ${TMP_DIR}/serverCert.pem -days 100000 -extensions SAN -extensions v3_req -extfile ${TMP_DIR}/server.conf
56+
57+
echo "Uploading certs to the cluster."
58+
kubectl create secret --namespace=kube-system generic vpa-tls-certs --from-file=${TMP_DIR}/caKey.pem --from-file=${TMP_DIR}/caCert.pem --from-file=${TMP_DIR}/serverKey.pem --from-file=${TMP_DIR}/serverCert.pem
59+
60+
if [ "${1:-unset}" = "e2e" ]; then
61+
openssl genrsa -out ${TMP_DIR}/e2eCaKey.pem 2048
62+
openssl req -x509 -new -nodes -key ${TMP_DIR}/e2eCaKey.pem -days 100000 -out ${TMP_DIR}/e2eCaCert.pem -subj "/CN=${CN_BASE}_e2e_ca" -addext "subjectAltName = DNS:${CN_BASE}_e2e_ca"
63+
openssl genrsa -out ${TMP_DIR}/e2eKey.pem 2048
64+
openssl req -new -key ${TMP_DIR}/e2eKey.pem -out ${TMP_DIR}/e2e.csr -subj "/CN=vpa-webhook.kube-system.svc" -config ${TMP_DIR}/server.conf
65+
openssl x509 -req -in ${TMP_DIR}/e2e.csr -CA ${TMP_DIR}/e2eCaCert.pem -CAkey ${TMP_DIR}/e2eCaKey.pem -CAcreateserial -out ${TMP_DIR}/e2eCert.pem -days 100000 -extensions SAN -extensions v3_req -extfile ${TMP_DIR}/server.conf
66+
echo "Uploading rotation e2e test certs to the cluster."
67+
kubectl create secret --namespace=kube-system generic vpa-e2e-certs --from-file=${TMP_DIR}/e2eCaKey.pem --from-file=${TMP_DIR}/e2eCaCert.pem --from-file=${TMP_DIR}/e2eKey.pem --from-file=${TMP_DIR}/e2eCert.pem
68+
fi
69+
70+
# Clean up after we're done.
71+
echo "Deleting ${TMP_DIR}."
72+
rm -rf ${TMP_DIR}

‎deploy/example/vpa/install-components/recommender-deployment.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
apiVersion: apps/v1
3+
kind: Deployment
4+
metadata:
5+
name: vpa-recommender
6+
namespace: kube-system
7+
spec:
8+
replicas: 1
9+
selector:
10+
matchLabels:
11+
app: vpa-recommender
12+
template:
13+
metadata:
14+
labels:
15+
app: vpa-recommender
16+
spec:
17+
serviceAccountName: vpa-recommender
18+
securityContext:
19+
runAsNonRoot: true
20+
runAsUser: 65534 # nobody
21+
containers:
22+
- name: recommender
23+
image: registry.k8s.io/autoscaling/vpa-recommender:1.3.0
24+
imagePullPolicy: IfNotPresent
25+
resources:
26+
limits:
27+
cpu: 200m
28+
memory: 1000Mi
29+
requests:
30+
cpu: 50m
31+
memory: 500Mi
32+
ports:
33+
- name: prometheus
34+
containerPort: 8942

‎deploy/example/vpa/install-components/rmcerts.sh‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/bin/bash
2+
3+
# Copyright 2018 The Kubernetes Authors.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# Generates the a CA cert, a server key, and a server cert signed by the CA.
18+
# reference:
19+
# https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/webhook/gencerts.sh
20+
set -e
21+
22+
echo "Deleting VPA Admission Controller certs."
23+
kubectl delete secret --namespace=kube-system vpa-tls-certs --ignore-not-found
24+
kubectl delete secret --namespace=kube-system vpa-e2e-certs --ignore-not-found

0 commit comments

Comments
 (0)