Skip to content

Commit 4384197

Browse files
mburke5678mburke5678
authored
Merge pull request #75005 from mburke5678/vpa-infra
Ability to schedule Vertical pod autoscaler pods on Infra/worker nodes
2 parents 3e54237 + 8fbfee3 commit 4384197

File tree

4 files changed

+369
-1
lines changed

4 files changed

+369
-1
lines changed

‎machine_management/creating-infrastructure-machinesets.adoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,8 @@ include::modules/infrastructure-moving-registry.adoc[leveloffset=+2]
128128

129129
include::modules/infrastructure-moving-monitoring.adoc[leveloffset=+2]
130130

131+
include::modules/nodes-pods-vertical-autoscaler-moving-vpa.adoc[leveloffset=+2]
132+
131133
[role="_additional-resources"]
132134
.Additional resources
133135
* xref:../observability/monitoring/configuring-the-monitoring-stack.adoc#moving-monitoring-components-to-different-nodes_configuring-the-monitoring-stack[Moving monitoring components to different nodes]

‎modules/nodes-pods-vertical-autoscaler-install.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ is automatically created if it does not exist.
3636

3737
.. Navigate to *Workloads* -> *Deployments* to verify that there are four deployments running.
3838

39-
. Optional. Verify the installation in the {product-title} CLI using the following command:
39+
. Verify the installation in the {product-title} CLI using the following command:
4040
+
4141
[source,terminal]
4242
----

‎modules/nodes-pods-vertical-autoscaler-moving-vpa.adoc

Lines changed: 360 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,360 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * machine_management/creating-infrastructure-machinesets.adoc
4+
// * nodes/pods/nodes-pods-vertical-autoscaler
5+
6+
ifeval::["{context}" == "nodes-pods-vertical-autoscaler"]
7+
:vpa:
8+
endif::[]
9+
ifeval::["{context}" == "creating-infrastructure-machinesets"]
10+
:machinemgmt:
11+
endif::[]
12+
13+
:_mod-docs-content-type: PROCEDURE
14+
[id="infrastructure-moving-vpa_{context}"]
15+
= Moving the Vertical Pod Autoscaler Operator components
16+
17+
ifdef::machinemgmt[]
18+
The Vertical Pod Autoscaler Operator (VPA) consists of three components: the recommender, updater, and admission controller. The Operator and each component has its own pod in the VPA namespace on the control plane nodes. You can move the VPA Operator and component pods to infrastructure nodes by adding a node selector to the VPA subscription and the `VerticalPodAutoscalerController` CR.
19+
endif::machinemgmt[]
20+
ifdef::vpa[]
21+
The Vertical Pod Autoscaler Operator (VPA) and each component has its own pod in the VPA namespace on the control plane nodes. You can move the VPA Operator and component pods to infrastructure or worker nodes by adding a node selector to the VPA subscription and the `VerticalPodAutoscalerController` CR.
22+
23+
You can create and use infrastructure nodes to host only infrastructure components, such as the default router, the integrated container image registry, and the components for cluster metrics and monitoring. These infrastructure nodes are not counted toward the total number of subscriptions that are required to run the environment. For more information, see _Creating infrastructure machine sets_.
24+
25+
You can move the components to the same node or separate nodes as appropriate for your organization.
26+
endif::vpa[]
27+
28+
The following example shows the default deployment of the VPA pods to the control plane nodes.
29+
30+
.Example output
31+
[source,terminal]
32+
----
33+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
34+
vertical-pod-autoscaler-operator-6c75fcc9cd-5pb6z 1/1 Running 0 7m59s 10.128.2.24 c416-tfsbj-master-1 <none> <none>
35+
vpa-admission-plugin-default-6cb78d6f8b-rpcrj 1/1 Running 0 5m37s 10.129.2.22 c416-tfsbj-master-1 <none> <none>
36+
vpa-recommender-default-66846bd94c-dsmpp 1/1 Running 0 5m37s 10.129.2.20 c416-tfsbj-master-0 <none> <none>
37+
vpa-updater-default-db8b58df-2nkvf 1/1 Running 0 5m37s 10.129.2.21 c416-tfsbj-master-1 <none> <none>
38+
----
39+
40+
.Procedure
41+
42+
ifdef::machinemgmt[]
43+
. Move the VPA Operator pod by adding a node selector to the `Subscription` custom resource (CR) for the VPA Operator:
44+
45+
.. Edit the CR:
46+
+
47+
[source,terminal]
48+
----
49+
$ oc edit Subscription vertical-pod-autoscaler -n openshift-vertical-pod-autoscaler
50+
----
51+
52+
.. Add a node selector to match the node role label on the infra node:
53+
+
54+
[source,terminal]
55+
----
56+
apiVersion: operators.coreos.com/v1alpha1
57+
kind: Subscription
58+
metadata:
59+
labels:
60+
operators.coreos.com/vertical-pod-autoscaler.openshift-vertical-pod-autoscaler: ""
61+
name: vertical-pod-autoscaler
62+
# ...
63+
spec:
64+
config:
65+
nodeSelector:
66+
node-role.kubernetes.io/infra: "" <1>
67+
----
68+
<1> Specifies the node role of an infra node.
69+
+
70+
[NOTE]
71+
====
72+
If the infra node uses taints, you need to add a toleration to the `Subscription` CR.
73+
74+
For example:
75+
76+
[source,terminal]
77+
----
78+
apiVersion: operators.coreos.com/v1alpha1
79+
kind: Subscription
80+
metadata:
81+
labels:
82+
operators.coreos.com/vertical-pod-autoscaler.openshift-vertical-pod-autoscaler: ""
83+
name: vertical-pod-autoscaler
84+
# ...
85+
spec:
86+
config:
87+
nodeSelector:
88+
node-role.kubernetes.io/infra: ""
89+
tolerations: <1>
90+
- key: "node-role.kubernetes.io/infra"
91+
operator: "Exists"
92+
effect: "NoSchedule"
93+
----
94+
====
95+
<1> Specifies a toleration for a taint on the infra node.
96+
97+
. Move each VPA component by adding node selectors to the `VerticalPodAutoscaler` custom resource (CR):
98+
99+
.. Edit the CR:
100+
+
101+
[source,terminal]
102+
----
103+
$ oc edit VerticalPodAutoscalerController default -n openshift-vertical-pod-autoscaler
104+
----
105+
106+
.. Add node selectors to match the node role label on the infra node:
107+
+
108+
[source,terminal]
109+
----
110+
apiVersion: autoscaling.openshift.io/v1
111+
kind: VerticalPodAutoscalerController
112+
metadata:
113+
name: default
114+
namespace: openshift-vertical-pod-autoscaler
115+
# ...
116+
spec:
117+
deploymentOverrides:
118+
admission:
119+
container:
120+
resources: {}
121+
nodeSelector:
122+
node-role.kubernetes.io/infra: "" <1>
123+
recommender:
124+
container:
125+
resources: {}
126+
nodeSelector:
127+
node-role.kubernetes.io/infra: "" <2>
128+
updater:
129+
container:
130+
resources: {}
131+
nodeSelector:
132+
node-role.kubernetes.io/infra: "" <3>
133+
----
134+
<1> Optional: Specifies the node role for the VPA admission pod.
135+
<2> Optional: Specifies the node role for the VPA recommender pod.
136+
<3> Optional: Specifies the node role for the VPA updater pod.
137+
+
138+
[NOTE]
139+
====
140+
If a target node uses taints, you need to add a toleration to the `VerticalPodAutoscalerController` CR.
141+
142+
For example:
143+
144+
[source,terminal]
145+
----
146+
apiVersion: autoscaling.openshift.io/v1
147+
kind: VerticalPodAutoscalerController
148+
metadata:
149+
name: default
150+
namespace: openshift-vertical-pod-autoscaler
151+
# ...
152+
spec:
153+
deploymentOverrides:
154+
admission:
155+
container:
156+
resources: {}
157+
nodeSelector:
158+
node-role.kubernetes.io/infra: ""
159+
tolerations: <1>
160+
- key: "my-example-node-taint-key"
161+
operator: "Exists"
162+
effect: "NoSchedule"
163+
recommender:
164+
container:
165+
resources: {}
166+
nodeSelector:
167+
node-role.kubernetes.io/infra: ""
168+
tolerations: <2>
169+
- key: "my-example-node-taint-key"
170+
operator: "Exists"
171+
effect: "NoSchedule"
172+
updater:
173+
container:
174+
resources: {}
175+
nodeSelector:
176+
node-role.kubernetes.io/infra: ""
177+
tolerations: <3>
178+
- key: "my-example-node-taint-key"
179+
operator: "Exists"
180+
effect: "NoSchedule"
181+
----
182+
====
183+
<1> Specifies a toleration for the admission controller pod for a taint on the infra node.
184+
<2> Specifies a toleration for the recommender pod for a taint on the infra node.
185+
<3> Specifies a toleration for the updater pod for a taint on the infra node.
186+
endif::machinemgmt[]
187+
188+
ifdef::vpa[]
189+
. Move the VPA Operator pod by adding a node selector to the `Subscription` custom resource (CR) for the VPA Operator:
190+
191+
.. Edit the CR:
192+
+
193+
[source,terminal]
194+
----
195+
$ oc edit Subscription vertical-pod-autoscaler -n openshift-vertical-pod-autoscaler
196+
----
197+
198+
.. Add a node selector to match the node role label on the node where you want to install the VPA Operator pod:
199+
+
200+
[source,terminal]
201+
----
202+
apiVersion: operators.coreos.com/v1alpha1
203+
kind: Subscription
204+
metadata:
205+
labels:
206+
operators.coreos.com/vertical-pod-autoscaler.openshift-vertical-pod-autoscaler: ""
207+
name: vertical-pod-autoscaler
208+
# ...
209+
spec:
210+
config:
211+
nodeSelector:
212+
node-role.kubernetes.io/<node_role>: "" <1>
213+
----
214+
<1> Specifies the node role of the node where you want to move the VPA Operator pod.
215+
+
216+
[NOTE]
217+
====
218+
If the infra node uses taints, you need to add a toleration to the `Subscription` CR.
219+
220+
For example:
221+
222+
[source,terminal]
223+
----
224+
apiVersion: operators.coreos.com/v1alpha1
225+
kind: Subscription
226+
metadata:
227+
labels:
228+
operators.coreos.com/vertical-pod-autoscaler.openshift-vertical-pod-autoscaler: ""
229+
name: vertical-pod-autoscaler
230+
# ...
231+
spec:
232+
config:
233+
nodeSelector:
234+
node-role.kubernetes.io/infra: ""
235+
tolerations: <1>
236+
- key: "node-role.kubernetes.io/infra"
237+
operator: "Exists"
238+
effect: "NoSchedule"
239+
----
240+
====
241+
<1> Specifies a toleration for a taint on the node where you want to move the VPA Operator pod.
242+
243+
. Move each VPA component by adding node selectors to the `VerticalPodAutoscaler` custom resource (CR):
244+
245+
.. Edit the CR:
246+
+
247+
[source,terminal]
248+
----
249+
$ oc edit VerticalPodAutoscalerController default -n openshift-vertical-pod-autoscaler
250+
----
251+
252+
.. Add node selectors to match the node role label on the node where you want to install the VPA components:
253+
+
254+
[source,terminal]
255+
----
256+
apiVersion: autoscaling.openshift.io/v1
257+
kind: VerticalPodAutoscalerController
258+
metadata:
259+
name: default
260+
namespace: openshift-vertical-pod-autoscaler
261+
# ...
262+
spec:
263+
deploymentOverrides:
264+
admission:
265+
container:
266+
resources: {}
267+
nodeSelector:
268+
node-role.kubernetes.io/<node_role>: "" <1>
269+
recommender:
270+
container:
271+
resources: {}
272+
nodeSelector:
273+
node-role.kubernetes.io/<node_role>: "" <2>
274+
updater:
275+
container:
276+
resources: {}
277+
nodeSelector:
278+
node-role.kubernetes.io/<node_role>: "" <3>
279+
----
280+
<1> Optional: Specifies the node role for the VPA admission pod.
281+
<2> Optional: Specifies the node role for the VPA recommender pod.
282+
<3> Optional: Specifies the node role for the VPA updater pod.
283+
+
284+
[NOTE]
285+
====
286+
If a target node uses taints, you need to add a toleration to the `VerticalPodAutoscalerController` CR.
287+
288+
For example:
289+
290+
[source,terminal]
291+
----
292+
apiVersion: autoscaling.openshift.io/v1
293+
kind: VerticalPodAutoscalerController
294+
metadata:
295+
name: default
296+
namespace: openshift-vertical-pod-autoscaler
297+
# ...
298+
spec:
299+
deploymentOverrides:
300+
admission:
301+
container:
302+
resources: {}
303+
nodeSelector:
304+
node-role.kubernetes.io/worker: ""
305+
tolerations: <1>
306+
- key: "my-example-node-taint-key"
307+
operator: "Exists"
308+
effect: "NoSchedule"
309+
recommender:
310+
container:
311+
resources: {}
312+
nodeSelector:
313+
node-role.kubernetes.io/worker: ""
314+
tolerations: <2>
315+
- key: "my-example-node-taint-key"
316+
operator: "Exists"
317+
effect: "NoSchedule"
318+
updater:
319+
container:
320+
resources: {}
321+
nodeSelector:
322+
node-role.kubernetes.io/worker: ""
323+
tolerations: <3>
324+
- key: "my-example-node-taint-key"
325+
operator: "Exists"
326+
effect: "NoSchedule"
327+
----
328+
====
329+
<1> Specifies a toleration for the admission controller pod for a taint on the node where you want to install the pod.
330+
<2> Specifies a toleration for the recommender pod for a taint on the node where you want to install the pod.
331+
<3> Specifies a toleration for the updater pod for a taint on the node where you want to install the pod.
332+
endif::vpa[]
333+
334+
.Verification
335+
336+
* You can verify the pods have moved by using the following command:
337+
+
338+
[source,terminal]
339+
----
340+
$ oc get pods -n openshift-vertical-pod-autoscaler -o wide
341+
----
342+
+
343+
The pods are no longer deployed to the control plane nodes.
344+
+
345+
.Example output
346+
[source,terminal]
347+
----
348+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
349+
vertical-pod-autoscaler-operator-6c75fcc9cd-5pb6z 1/1 Running 0 7m59s 10.128.2.24 c416-tfsbj-infra-eastus3-2bndt <none> <none>
350+
vpa-admission-plugin-default-6cb78d6f8b-rpcrj 1/1 Running 0 5m37s 10.129.2.22 c416-tfsbj-infra-eastus1-lrgj8 <none> <none>
351+
vpa-recommender-default-66846bd94c-dsmpp 1/1 Running 0 5m37s 10.129.2.20 c416-tfsbj-infra-eastus1-lrgj8 <none> <none>
352+
vpa-updater-default-db8b58df-2nkvf 1/1 Running 0 5m37s 10.129.2.21 c416-tfsbj-infra-eastus1-lrgj8 <none> <none>
353+
----
354+
355+
ifeval::["{context}" == "nodes-pods-vertical-autoscaler"]
356+
:!vpa:
357+
endif::[]
358+
ifeval::["{context}" == "creating-infrastructure-machinesets"]
359+
:!machinemgmt:
360+
endif::[]

‎nodes/pods/nodes-pods-vertical-autoscaler.adoc

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,12 @@ include::modules/nodes-pods-vertical-autoscaler-about.adoc[leveloffset=+1]
2222

2323
include::modules/nodes-pods-vertical-autoscaler-install.adoc[leveloffset=+1]
2424

25+
include::modules/nodes-pods-vertical-autoscaler-moving-vpa.adoc[leveloffset=+1]
26+
27+
.Additional resources
28+
29+
* xref:../../machine_management/creating-infrastructure-machinesets.adoc#creating-infrastructure-machinesets-production[Creating infrastructure machine sets]
30+
2531
include::modules/nodes-pods-vertical-autoscaler-using-about.adoc[leveloffset=+1]
2632

2733
include::modules/nodes-pods-vertical-autoscaler-tuning.adoc[leveloffset=+2]

0 commit comments

Comments
 (0)