Skip to content

Commit 3f6e5b8

Browse files
stevsmitSteven Smith
stevsmit
and
Steven Smith
authored
Updates infra proc for Quay (quay#1439)
Co-authored-by: Steven Smith <[email protected]>
1 parent 3096ee9 commit 3f6e5b8

File tree

7 files changed

+354
-157
lines changed

7 files changed

+354
-157
lines changed

‎deploy_red_hat_quay_operator/master.adoc‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,11 +67,18 @@ include::modules/first-user-api.adoc[leveloffset=+3]
6767
include::modules/operator-deploy-view-pods-cli.adoc[leveloffset=+3]
6868
include::modules/operator-deploy-hpa.adoc[leveloffset=+3]
6969

70+
//infrastructure
71+
include::modules/operator-deploy-infrastructure.adoc[leveloffset=+1]
72+
include::modules/labeling-taint-nodes-for-infrastructure-use.adoc[leveloffset=+2]
73+
include::modules/creating-project-node-selector-toleration.adoc[leveloffset=+2]
74+
include::modules/installing-quay-operator-namespace.adoc[leveloffset=+2]
75+
include::modules/creating-registry-infra-node.adoc[leveloffset=+2]
76+
7077
[role="_additional-resources"]
7178
.Additional resources
7279
For more information on pre-configuring your {productname} deployment, see the section xref:config-preconfigure-automation[Pre-configuring {productname} for automation]
7380

74-
include::modules/operator-monitor-deploy-cli.adoc[leveloffset=+3]
81+
include::modules/operator-monitor-deploy-cli.adoc[leveloffset=+1]
7582
//ui
7683
include::modules/operator-deploy-ui.adoc[leveloffset=+2]
7784
include::modules/operator-first-user-ui.adoc[leveloffset=+3]

‎modules/creating-project-node-selector-toleration.adoc‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
2+
[id="creating-project-node-selector-toleration"]
3+
= Creating a project with node selector and tolerations
4+
5+
Use the following procedure to create a project with the `node-selector` and `tolerations` annotations.
6+
7+
.Procedure
8+
9+
. Add the `node-selector` annotation to the namespace by entering the following command:
10+
+
11+
[source,terminal]
12+
----
13+
$ oc annotate namespace <namespace> openshift.io/node-selector='node-role.kubernetes.io/infra='
14+
----
15+
+
16+
.Example output
17+
+
18+
[source,yaml]
19+
----
20+
namespace/<namespace> annotated
21+
----
22+
23+
. Add the `tolerations` annotation to the namespace by entering the following command:
24+
+
25+
[source,terminal]
26+
----
27+
$ oc annotate namespace <namespace> scheduler.alpha.kubernetes.io/defaultTolerations='[{"operator":"Equal","value":"reserved","effect":"NoSchedule","key":"node-role.kubernetes.io/infra"},{"operator":"Equal","value":"reserved","effect":"NoExecute","key":"node-role.kubernetes.io/infra"}]' --overwrite
28+
----
29+
+
30+
.Example output
31+
+
32+
[source,yaml]
33+
----
34+
namespace/<namespace> annotated
35+
----
36+
+
37+
[IMPORTANT]
38+
====
39+
The tolerations in this example are specific to two taints commonly applied to infra nodes. The taints configured in your environment might differ. You must set the tolerations accordingly to match the taints applied to your infra nodes.
40+
====

‎modules/creating-registry-infra-node.adoc‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
2+
[id="creating-registry-infra-node"]
3+
= Creating the {productname} registry
4+
5+
After you have downloaded the {productname} Operator, you must create the {productname} registry. The registry's components, for example, `clair`, `postgres`, `redis`, and so on, must be patched with the `toleration` annotation so that they can schedule onto the `infra` worker nodes.
6+
7+
The following procedure shows you how to create a {productname} registry that runs on infrastructure nodes.
8+
9+
.Procedure
10+
11+
. On the {ocp} web console, click *Operators* -> *Installed Operators* -> *Red Hat Quay*.
12+
13+
. On the *{productname} Operator details* page, click *Quay Registry* -> *Create QuayRegistry*.
14+
15+
. On the *Create QuayRegistry* page, set the `monitoring` and `objectstorage` fields to `false`. The monitoring component cannot be enabled when {productname} is installed in a single namespace. For example:
16+
+
17+
[source,yaml]
18+
----
19+
# ...
20+
- kind: monitoring
21+
managed: false
22+
- kind: objectstorage
23+
managed: false
24+
# ...
25+
----
26+
27+
. Click *Create*.
28+
29+
////
30+
. The following condition is reported: `Condition: RolloutBlocked`. This occurs because all pods for the registry must include the `node-role.kubernetes.io/infra` nodeSelector and toleration. Apply the `node-role.kubernetes.io/infra` nodeSelector and toleration to all pods by entering the following command:
31+
+
32+
[source,terminal]
33+
----
34+
$ for deploy in $(oc get deployments -n <annotated_namespace> -o name | grep -E 'example-registry-(clair|quay)'); do
35+
oc patch $deploy -n annotated_namespace --type='strategic' -p '{
36+
"spec": {
37+
"template": {
38+
"spec": {
39+
"nodeSelector": {
40+
"node-role.kubernetes.io/infra": ""
41+
},
42+
"tolerations": [
43+
{
44+
"key": "node-role.kubernetes.io/infra",
45+
"operator": "Exists",
46+
"effect": "NoSchedule"
47+
}
48+
]
49+
}
50+
}
51+
}
52+
}'
53+
done
54+
----
55+
+
56+
.Example output
57+
+
58+
[source,terminal]
59+
----
60+
deployment.apps/example-registry-clair-app patched
61+
deployment.apps/example-registry-clair-postgres patched
62+
deployment.apps/example-registry-quay-app patched
63+
deployment.apps/example-registry-quay-database patched
64+
deployment.apps/example-registry-quay-mirror patched
65+
deployment.apps/example-registry-quay-redis patched
66+
----
67+
68+
. Ensure that all pods include the `node-role.kubernetes.io/infra` nodeSelector and toleration by entering the following command:
69+
+
70+
[source,terminal]
71+
----
72+
$ for deploy in $(oc get deployments -n <annotated_namespace> -o name | grep example-registry); do
73+
echo $deploy
74+
oc get -n <annotated_namespace> $deploy -o yaml | grep -A5 nodeSelector
75+
oc get -n <annotated_namespace> $deploy -o yaml | grep -A5 tolerations
76+
done
77+
----
78+
+
79+
.Example output
80+
+
81+
[source,terminal]
82+
----
83+
...
84+
example-registry-clair-app
85+
nodeSelector:
86+
node-role.kubernetes.io/infra: ""
87+
restartPolicy: Always
88+
schedulerName: default-scheduler
89+
securityContext: {}
90+
serviceAccount: example-registry-clair-app
91+
tolerations:
92+
- effect: NoSchedule
93+
key: node-role.kubernetes.io/infra
94+
operator: Exists
95+
volumes:
96+
- configMap:
97+
...
98+
----
99+
////
100+
101+
. Optional: Confirm that the pods are running on infra nodes.
102+
103+
.. List all `Quay`-related pods along with the nodes that they are scheduled on by entering the following command:
104+
+
105+
[source,terminal]
106+
----
107+
$ oc get pods -n <annotated_namespace> -o wide | grep example-registry
108+
----
109+
+
110+
.Example output
111+
+
112+
[source,terminal]
113+
----
114+
...
115+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
116+
example-registry-clair-app-5f95d685bd-dgjf6 1/1 Running 0 52m 10.128.4.12 example-cluster-new-c5qqp-worker-b-wrhw4.c.quay-devel.internal <none> <none>
117+
...
118+
----
119+
120+
.. Confirm that the nodes listed include only nodes labeled `infra` by running the following command:
121+
+
122+
[source,terminal]
123+
----
124+
$ oc get nodes -l node-role.kubernetes.io/infra -o name
125+
----
126+
+
127+
.Example output
128+
+
129+
[source,terminal]
130+
----
131+
node/example-cluster-new-c5qqp-worker-b-4zxx5.c.quay-devel.internal modified
132+
node/example-cluster-new-c5qqp-worker-b-kz6jn.c.quay-devel.internal modified
133+
node/example-cluster-new-c5qqp-worker-b-wrhw4.c.quay-devel.internal modified
134+
----
135+
+
136+
[NOTE]
137+
====
138+
If any pod appears on a non-infra node, revisit your namespace annotations and deployment patching.
139+
====
140+
141+
. Restart all pods for the {productname} registry by entering the following command:
142+
+
143+
[source,terminal]
144+
----
145+
$ oc delete pod -n <annotated_namespace> --all
146+
----
147+
148+
. Check the status of the pods by entering the following command:
149+
+
150+
[source,terminal]
151+
----
152+
$ oc get pods -n <annotated_namespace>
153+
----
154+
+
155+
.Example output
156+
+
157+
[source,terminal]
158+
----
159+
...
160+
NAME READY STATUS RESTARTS AGE
161+
example-registry-clair-app-5f95d685bd-dgjf6 1/1 Running 0 5m4s
162+
...
163+
----

‎modules/installing-quay-operator-namespace.adoc‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
2+
[id="installing-quay-operator-namespace"]
3+
= Installing the {productname} Operator on the annotated namespace
4+
5+
After you have added the `node-role.kubernetes.io/infra=` label to worker nodes and added the `node-selector` and `tolerations` annotations to the namespace, you must download the {productname} Operator in that namespace.
6+
7+
The following procedure shows you how to download the {productname} Operator on the annotated namespace and how to update the subscription to ensure successful installation.
8+
9+
.Procedure
10+
11+
. On the {ocp} web console, click *Operators* -> *OperatorHub*.
12+
13+
. In the search box, type *{productname}*.
14+
15+
. Click *{productname}* -> *Install*.
16+
17+
. Select the update channel, for example, *stable-{producty}* and the version.
18+
19+
. Click *A specific namespace on the cluster* for the installation mode, and then select the namespace that you applied the `node-selector` and `tolerations` annotations to.
20+
21+
. Click *Install*.
22+
////
23+
. After a few minutes, the {productname} Operator installation fails. This occurs because the Operator itself must run on the `infra` nodes. Update the {productname} Operator subscription to run on the infra nodes by entering the following command:
24+
+
25+
[source,terminal]
26+
----
27+
$ oc patch subscription quay-operator -n <annotated_namespace> \
28+
--type=merge -p '{
29+
"spec": {
30+
"config": {
31+
"nodeSelector": {"node-role.kubernetes.io/infra": ""},
32+
"tolerations": [
33+
{"key":"node-role.kubernetes.io/infra","operator":"Exists","effect":"NoSchedule"}
34+
]
35+
}
36+
}
37+
}'
38+
----
39+
+
40+
The Operator resumes downloading.
41+
////
42+
43+
. Confirm that the Operator is installed by entering the following command:
44+
+
45+
[source,terminal]
46+
----
47+
$ oc get pods -n <annotated_namespace> -o wide | grep quay-operator
48+
----
49+
+
50+
.Example output
51+
+
52+
[source,terminal]
53+
----
54+
quay-operator.v3.15.1-858b5c5fdc-lf5kj 1/1 Running 0 29m 10.130.6.18 example-cluster-new-c5qqp-worker-f-mhngl.c.quay-devel.internal <none> <none>
55+
----

‎modules/labeling-taint-nodes-for-infrastructure-use.adoc‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
[id="labeling-taint-nodes-for-infrastructure-use"]
2+
= Labeling and tainting nodes for infrastructure use
3+
4+
Use the following procedure to label and taint nodes for infrastructure use.
5+
6+
[NOTE]
7+
====
8+
The following procedure labels three worker nodes with the `infra` label. Depending on the resources relevant to your environment, you might have to label more than three worker nodes with the `infra` label.
9+
====
10+
11+
. Obtain a list of _worker_ nodes in your deployment by entering the following command:
12+
+
13+
[source,terminal]
14+
----
15+
$ oc get nodes | grep worker
16+
----
17+
+
18+
.Example output
19+
+
20+
[source,terminal]
21+
----
22+
NAME STATUS ROLES AGE VERSION
23+
---
24+
example-cluster-new-c5qqp-worker-b-4zxx5.c.quay-devel.internal Ready worker 401d v1.31.11
25+
example-cluster-new-c5qqp-worker-b-kz6jn.c.quay-devel.internal Ready worker 402d v1.31.11
26+
example-cluster-new-c5qqp-worker-b-wrhw4.c.quay-devel.internal Ready worker 401d v1.31.11
27+
---
28+
----
29+
30+
. Add the `node-role.kubernetes.io/infra=` label to the worker nodes by entering the following command.
31+
The number of infrastructure nodes required depends on your environment. Production environments should provision enough infra nodes to ensure high availability and sufficient resources for all `quay`-related components. Monitor CPU, memory, and storage utilization to determine if additional infra nodes are required.
32+
+
33+
[source,terminal]
34+
----
35+
$ oc label node --overwrite <infra_node_one> <infra_node_two> <infra_node_three> node-role.kubernetes.io/infra=
36+
----
37+
38+
. Confirm that the `node-role.kubernetes.io/infra=` label has been added to the proper nodes by entering the following command:
39+
+
40+
[source,terminal]
41+
----
42+
$ oc get node | grep infra
43+
----
44+
+
45+
[source,terminal]
46+
----
47+
---
48+
example-cluster-new-c5qqp-worker-b-4zxx5.c.quay-devel.internal Ready infra,worker 405d v1.32.8
49+
example-cluster-new-c5qqp-worker-b-kz6jn.c.quay-devel.internal Ready infra,worker 406d v1.32.8
50+
example-cluster-new-c5qqp-worker-b-wrhw4.c.quay-devel.internal Ready infra,worker 405d v1.32.8
51+
---
52+
----
53+
54+
. When a worker node is assigned the `infra` role, there is a chance that user workloads could get inadvertently assigned to an infra node. To avoid this, you can apply a taint to the infra node, and then add tolerations for the pods that you want to control. Taint the worker nodes with the `infra` label by entering the following command:
55+
+
56+
[source,terminal]
57+
----
58+
$ oc adm taint nodes -l node-role.kubernetes.io/infra \
59+
node-role.kubernetes.io/infra=reserved:NoSchedule --overwrite
60+
----
61+
+
62+
.Example output
63+
+
64+
[source,terminal]
65+
----
66+
node/example-cluster-new-c5qqp-worker-b-4zxx5.c.quay-devel.internal modified
67+
node/example-cluster-new-c5qqp-worker-b-kz6jn.c.quay-devel.internal modified
68+
node/example-cluster-new-c5qqp-worker-b-wrhw4.c.quay-devel.internal modified
69+
----

0 commit comments

Comments
 (0)