Skip to content

Commit 4604cf5

Browse files
mburke5678mburke5678
authored
Merge pull request #80822 from mburke5678/podauto-cro-infra
Ability to run CRO on infra/worker node
2 parents 5b791d0 + cdee176 commit 4604cf5

6 files changed

+235
-12
lines changed

‎machine_management/creating-infrastructure-machinesets.adoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,8 @@ include::modules/infrastructure-moving-monitoring.adoc[leveloffset=+2]
125125

126126
include::modules/nodes-pods-vertical-autoscaler-moving-vpa.adoc[leveloffset=+2]
127127

128+
include::modules/nodes-cluster-resource-override-move-infra.adoc[leveloffset=+2]
129+
128130
[role="_additional-resources"]
129131
.Additional resources
130132
* xref:../observability/monitoring/configuring-the-monitoring-stack.adoc#moving-monitoring-components-to-different-nodes_configuring-the-monitoring-stack[Moving monitoring components to different nodes]

‎modules/nodes-cluster-resource-configure.adoc

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@
1111
The Cluster Resource Override Operator requires a `ClusterResourceOverride` custom resource (CR)
1212
and a label for each project where you want the Operator to control overcommit.
1313

14+
ifndef::openshift-rosa,openshift-dedicated[]
15+
By default, the installation process creates two Cluster Resource Override pods on the control plane nodes in the `clusterresourceoverride-operator` namespace. You can move these pods to other nodes, such as infrastructure nodes, as needed. Infrastructure nodes are not counted toward the total number of subscriptions that are required to run the environment. For more information, see "Moving the Cluster Resource Override Operator pods".
16+
endif::openshift-rosa,openshift-dedicated[]
17+
1418
.Prerequisites
1519

1620
* The Cluster Resource Override Operator has no effect if limits have not
@@ -36,9 +40,9 @@ spec:
3640
limitCPUToMemoryPercent: 200 <3>
3741
# ...
3842
----
39-
<1> Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.
40-
<2> Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.
41-
<3> Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.
43+
<1> Optional: Specify the percentage to override the container memory limit, if used, between 1-100. The default is `50`.
44+
<2> Optional: Specify the percentage to override the container CPU limit, if used, between 1-100. The default is `25`.
45+
<3> Optional: Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed before overriding the CPU request, if configured. The default is `200`.
4246

4347
. Ensure the following label has been added to the Namespace object for each project where you want the Cluster Resource Override Operator to control overcommit:
4448
+

‎modules/nodes-cluster-resource-override-deploy-cli.adoc

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,11 @@
77
[id="nodes-cluster-resource-override-deploy-cli_{context}"]
88
= Installing the Cluster Resource Override Operator using the CLI
99

10-
You can use the {product-title} CLI to install the Cluster Resource Override Operator to help control overcommit in your cluster.
10+
You can use the {product-title} CLI to install the Cluster Resource Override Operator to help control overcommit in your cluster.
11+
12+
ifndef::openshift-rosa,openshift-dedicated[]
13+
By default, the installation process creates a Cluster Resource Override Operator pod on a worker node in the `clusterresourceoverride-operator` namespace. You can move this pod to another node, such as an infrastructure node, as needed. Infrastructure nodes are not counted toward the total number of subscriptions that are required to run the environment. For more information, see "Moving the Cluster Resource Override Operator pods".
14+
endif::openshift-rosa,openshift-dedicated[]
1115

1216
.Prerequisites
1317

@@ -130,9 +134,9 @@ spec:
130134
limitCPUToMemoryPercent: 200 <4>
131135
----
132136
<1> The name must be `cluster`.
133-
<2> Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.
134-
<3> Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.
135-
<4> Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.
137+
<2> Optional: Specify the percentage to override the container memory limit, if used, between 1-100. The default is `50`.
138+
<3> Optional: Specify the percentage to override the container CPU limit, if used, between 1-100. The default is `25`.
139+
<4> Optional: Specify the percentage to override the container memory limit, if used. Scaling 1 Gi of RAM at 100 percent is equal to 1 CPU core. This is processed before overriding the CPU request, if configured. The default is `200`.
136140

137141
.. Create the `ClusterResourceOverride` object:
138142
+

‎modules/nodes-cluster-resource-override-deploy-console.adoc

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,11 @@
66
[id="nodes-cluster-resource-override-deploy-console_{context}"]
77
= Installing the Cluster Resource Override Operator using the web console
88

9-
You can use the {product-title} web console to install the Cluster Resource Override Operator to help control overcommit in your cluster.
9+
You can use the {product-title} CLI to install the Cluster Resource Override Operator to help control overcommit in your cluster.
10+
11+
ifndef::openshift-rosa,openshift-dedicated[]
12+
By default, the installation process creates a Cluster Resource Override Operator pod on a worker node in the `clusterresourceoverride-operator` namespace. You can move this pod to another node, such as an infrastructure node, as needed. Infrastructure nodes are not counted toward the total number of subscriptions that are required to run the environment. For more information, see "Moving the Cluster Resource Override Operator pods".
13+
endif::openshift-rosa,openshift-dedicated[]
1014

1115
.Prerequisites
1216

@@ -55,12 +59,11 @@ spec:
5559
memoryRequestToLimitPercent: 50 <2>
5660
cpuRequestToLimitPercent: 25 <3>
5761
limitCPUToMemoryPercent: 200 <4>
58-
# ...
5962
----
6063
<1> The name must be `cluster`.
61-
<2> Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.
62-
<3> Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.
63-
<4> Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.
64+
<2> Optional: Specify the percentage to override the container memory limit, if used, between 1-100. The default is `50`.
65+
<3> Optional: Specify the percentage to override the container CPU limit, if used, between 1-100. The default is `25`.
66+
<4> Optional: Specify the percentage to override the container memory limit, if used. Scaling 1 Gi of RAM at 100 percent is equal to 1 CPU core. This is processed before overriding the CPU request, if configured. The default is `200`.
6467

6568
.. Click *Create*.
6669

‎modules/nodes-cluster-resource-override-move-infra.adoc

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * nodes/clusters/nodes-cluster-overcommit.adoc
4+
// * machine_management/creating-infrastructure-machinesets.adoc
5+
6+
ifeval::["{context}" == "nodes-cluster-overcommit"]
7+
:cro:
8+
endif::[]
9+
10+
:_mod-docs-content-type: PROCEDURE
11+
[id="nodes-cluster-resource-override-move-infra_{context}"]
12+
= Moving the Cluster Resource Override Operator pods
13+
14+
By default, the Cluster Resource Override Operator installation process creates an Operator pod and two Cluster Resource Override pods on nodes in the `clusterresourceoverride-operator` namespace. You can move these pods to other nodes, such as infrastructure nodes, as needed.
15+
16+
ifdef::cro[]
17+
You can create and use infrastructure nodes to host only infrastructure components, such as the default router, the integrated container image registry, and the components for cluster metrics and monitoring. These infrastructure nodes are not counted toward the total number of subscriptions that are required to run the environment. For more information about infrastructure nodes, see "Creating infrastructure machine sets".
18+
endif::cro[]
19+
20+
The following examples shows the Cluster Resource Override pods are deployed to control plane nodes and the Cluster Resource Override Operator pod is deployed to a worker node.
21+
22+
.Example Cluster Resource Override pods
23+
[source,terminal]
24+
----
25+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
26+
clusterresourceoverride-786b8c898c-9wrdq 1/1 Running 0 23s 10.128.2.32 ip-10-0-14-183.us-west-2.compute.internal <none> <none>
27+
clusterresourceoverride-786b8c898c-vn2lf 1/1 Running 0 26s 10.130.2.10 ip-10-0-20-140.us-west-2.compute.internal <none> <none>
28+
clusterresourceoverride-operator-6b8b8b656b-lvr62 1/1 Running 0 56m 10.131.0.33 ip-10-0-2-39.us-west-2.compute.internal <none> <none>
29+
----
30+
31+
.Example node list
32+
[source,terminal]
33+
----
34+
NAME STATUS ROLES AGE VERSION
35+
ip-10-0-14-183.us-west-2.compute.internal Ready control-plane,master 65m v1.30.4
36+
ip-10-0-2-39.us-west-2.compute.internal Ready worker 58m v1.30.4
37+
ip-10-0-20-140.us-west-2.compute.internal Ready control-plane,master 65m v1.30.4
38+
ip-10-0-23-244.us-west-2.compute.internal Ready infra 55m v1.30.4
39+
ip-10-0-77-153.us-west-2.compute.internal Ready control-plane,master 65m v1.30.4
40+
ip-10-0-99-108.us-west-2.compute.internal Ready worker 24m v1.30.4
41+
ip-10-0-24-233.us-west-2.compute.internal Ready infra 55m v1.30.4
42+
ip-10-0-88-109.us-west-2.compute.internal Ready worker 24m v1.30.4
43+
ip-10-0-67-453.us-west-2.compute.internal Ready infra 55m v1.30.4
44+
----
45+
46+
.Procedure
47+
48+
. Move the Cluster Resource Override Operator pod by adding a node selector to the `Subscription` custom resource (CR) for the Cluster Resource Override Operator.
49+
50+
.. Edit the CR:
51+
+
52+
[source,terminal]
53+
----
54+
$ oc edit -n clusterresourceoverride-operator subscriptions.operators.coreos.com clusterresourceoverride
55+
----
56+
57+
.. Add a node selector to match the node role label on the node where you want to install the Cluster Resource Override Operator pod:
58+
+
59+
[source,terminal]
60+
----
61+
apiVersion: operators.coreos.com/v1alpha1
62+
kind: Subscription
63+
metadata:
64+
name: clusterresourceoverride
65+
namespace: clusterresourceoverride-operator
66+
# ...
67+
spec:
68+
config:
69+
nodeSelector:
70+
node-role.kubernetes.io/infra: "" <1>
71+
# ...
72+
----
73+
<1> Specify the role of the node where you want to deploy the Cluster Resource Override Operator pod.
74+
+
75+
[NOTE]
76+
====
77+
If the infra node uses taints, you need to add a toleration to the `Subscription` CR.
78+
79+
For example:
80+
81+
[source,terminal]
82+
----
83+
apiVersion: operators.coreos.com/v1alpha1
84+
kind: Subscription
85+
metadata:
86+
name: clusterresourceoverride
87+
namespace: clusterresourceoverride-operator
88+
# ...
89+
spec:
90+
config:
91+
nodeSelector:
92+
node-role.kubernetes.io/infra: ""
93+
tolerations: <1>
94+
- key: "node-role.kubernetes.io/infra"
95+
operator: "Exists"
96+
effect: "NoSchedule"
97+
----
98+
<1> Specifies a toleration for a taint on the infra node.
99+
====
100+
101+
. Move the Cluster Resource Override pods by adding a node selector to the `ClusterResourceOverride` custom resource (CR):
102+
103+
.. Edit the CR:
104+
+
105+
[source,terminal]
106+
----
107+
$ oc edit ClusterResourceOverride cluster -n clusterresourceoverride-operator
108+
----
109+
110+
.. Add a node selector to match the node role label on the infra node:
111+
+
112+
[source,terminal]
113+
----
114+
apiVersion: operator.autoscaling.openshift.io/v1
115+
kind: ClusterResourceOverride
116+
metadata:
117+
name: cluster
118+
resourceVersion: "37952"
119+
spec:
120+
podResourceOverride:
121+
spec:
122+
cpuRequestToLimitPercent: 25
123+
limitCPUToMemoryPercent: 200
124+
memoryRequestToLimitPercent: 50
125+
deploymentOverrides:
126+
replicas: 1 <1>
127+
nodeSelector:
128+
node-role.kubernetes.io/infra: "" <2>
129+
# ...
130+
----
131+
<1> Optional: Specify the number of Cluster Resource Override pods to deploy. The default is `2`. Only one pod is allowed per node.
132+
<2> Optional: Specify the role of the node where you want to deploy the Cluster Resource Override pods.
133+
+
134+
[NOTE]
135+
====
136+
If the infra node uses taints, you need to add a toleration to the `ClusterResourceOverride` CR.
137+
138+
For example:
139+
140+
[source,terminal]
141+
----
142+
apiVersion: operator.autoscaling.openshift.io/v1
143+
kind: ClusterResourceOverride
144+
metadata:
145+
name: cluster
146+
# ...
147+
spec:
148+
podResourceOverride:
149+
spec:
150+
memoryRequestToLimitPercent: 50
151+
cpuRequestToLimitPercent: 25
152+
limitCPUToMemoryPercent: 200
153+
deploymentOverrides:
154+
replicas: 3
155+
nodeSelector:
156+
node-role.kubernetes.io/worker: ""
157+
tolerations: <1>
158+
- key: "key"
159+
operator: "Equal"
160+
value: "value"
161+
effect: "NoSchedule"
162+
----
163+
<1> Specifies a toleration for a taint on the infra node.
164+
====
165+
166+
.Verification
167+
168+
* You can verify that the pods have moved by using the following command:
169+
+
170+
[source,terminal]
171+
----
172+
$ oc get pods -n clusterresourceoverride-operator -o wide
173+
----
174+
+
175+
The Cluster Resource Override pods are now deployed to the infra nodes.
176+
+
177+
.Example output
178+
[source,terminal]
179+
----
180+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
181+
clusterresourceoverride-786b8c898c-9wrdq 1/1 Running 0 23s 10.127.2.25 ip-10-0-23-244.us-west-2.compute.internal <none> <none>
182+
clusterresourceoverride-786b8c898c-vn2lf 1/1 Running 0 26s 10.128.0.80 ip-10-0-24-233.us-west-2.compute.internal <none> <none>
183+
clusterresourceoverride-operator-6b8b8b656b-lvr62 1/1 Running 0 56m 10.129.0.71 ip-10-0-67-453.us-west-2.compute.internal <none> <none>
184+
----
185+
186+
ifeval::["{context}" == "nodes-pods-vertical-autoscaler"]
187+
:!cro:
188+
endif::[]

‎nodes/clusters/nodes-cluster-overcommit.adoc

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,10 +58,32 @@ include::modules/nodes-cluster-resource-override.adoc[leveloffset=+1]
5858

5959
include::modules/nodes-cluster-resource-override-deploy-console.adoc[leveloffset=+2]
6060

61+
ifndef::openshift-rosa,openshift-dedicated[]
62+
.Additional resources
63+
* xref:../../machine_management/creating-infrastructure-machinesets.adoc#nodes-cluster-resource-override-move-infra_creating-infrastructure-machinesets[Moving the Cluster Resource Override Operator pods]
64+
endif::openshift-rosa,openshift-dedicated[]
65+
6166
include::modules/nodes-cluster-resource-override-deploy-cli.adoc[leveloffset=+2]
6267

68+
ifndef::openshift-rosa,openshift-dedicated[]
69+
.Additional resources
70+
* xref:../../machine_management/creating-infrastructure-machinesets.adoc#nodes-cluster-resource-override-move-infra_creating-infrastructure-machinesets[Moving the Cluster Resource Override Operator pods]
71+
endif::openshift-rosa,openshift-dedicated[]
72+
6373
include::modules/nodes-cluster-resource-configure.adoc[leveloffset=+2]
6474

75+
ifndef::openshift-rosa,openshift-dedicated[]
76+
.Additional resources
77+
* xref:../../machine_management/creating-infrastructure-machinesets.adoc#nodes-cluster-resource-override-move-infra_creating-infrastructure-machinesets[Moving the Cluster Resource Override Operator pods]
78+
endif::openshift-rosa,openshift-dedicated[]
79+
80+
ifndef::openshift-rosa,openshift-dedicated[]
81+
include::modules/nodes-cluster-resource-override-move-infra.adoc[leveloffset=+2]
82+
83+
.Additional resources
84+
* xref:../../machine_management/creating-infrastructure-machinesets.adoc#creating-infrastructure-machinesets[Creating infrastructure machine sets]
85+
endif::openshift-rosa,openshift-dedicated[]
86+
6587
include::modules/nodes-cluster-node-overcommit.adoc[leveloffset=+1]
6688

6789
include::modules/nodes-cluster-overcommit-resources-containers.adoc[leveloffset=+2]

0 commit comments

Comments
 (0)