Skip to content

Commit 60f33d9

Browse files
mjpytlakmjpytlak
authored
Merge pull request #57592 from tmalove/etcd-ocpbugs-3829-tlove-new
[OCPBUGS-3829]: Add etcd module from KCS article
2 parents 4ce4f7e + 62ad012 commit 60f33d9

File tree

2 files changed

+196
-0
lines changed

2 files changed

+196
-0
lines changed

‎modules/move-etcd-different-disk.adoc

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/recommended-host-practices.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="move-etcd-different-disk_{context}"]
7+
= Moving etcd to a different disk
8+
9+
You can move etcd from a shared disk to a separate disk to prevent or resolve performance issues.
10+
11+
.Prerequisites
12+
13+
* The `MachineConfigPool` must match `metadata.labels[machineconfiguration.openshift.io/role]`. This applies to a controller, worker, or a custom pool.
14+
* The node's auxiliary storage device, such as `/dev/sdb`, must match the sdb. Change this reference in all places in the file.
15+
16+
[NOTE]
17+
====
18+
This procedure does not move parts of the root file system, such as `/var/`, to another disk or partition on an installed node.
19+
====
20+
21+
The Machine Config Operator (MCO) is responsible for mounting a secondary disk for an {product-title} {product-version} container storage.
22+
23+
Use the following steps to move etcd to a different device:
24+
25+
.Procedure
26+
. Create a `machineconfig` YAML file named `etcd-mc.yml` and add the following information:
27+
+
28+
[source,yaml]
29+
----
30+
apiVersion: machineconfiguration.openshift.io/v1
31+
kind: MachineConfig
32+
metadata:
33+
labels:
34+
machineconfiguration.openshift.io/role: master
35+
name: 98-var-lib-etcd
36+
spec:
37+
config:
38+
ignition:
39+
version: 3.2.0
40+
systemd:
41+
units:
42+
- contents: |
43+
[Unit]
44+
Description=Make File System on /dev/sdb
45+
DefaultDependencies=no
46+
BindsTo=dev-sdb.device
47+
After=dev-sdb.device var.mount
48+
[email protected]
49+
50+
[Service]
51+
Type=oneshot
52+
RemainAfterExit=yes
53+
ExecStart=/usr/lib/systemd/systemd-makefs xfs /dev/sdb
54+
TimeoutSec=0
55+
56+
[Install]
57+
WantedBy=var-lib-containers.mount
58+
enabled: true
59+
name: [email protected]
60+
- contents: |
61+
[Unit]
62+
Description=Mount /dev/sdb to /var/lib/etcd
63+
Before=local-fs.target
64+
[email protected]
65+
[email protected] var.mount
66+
67+
[Mount]
68+
What=/dev/sdb
69+
Where=/var/lib/etcd
70+
Type=xfs
71+
Options=defaults,prjquota
72+
73+
[Install]
74+
WantedBy=local-fs.target
75+
enabled: true
76+
name: var-lib-etcd.mount
77+
- contents: |
78+
[Unit]
79+
Description=Sync etcd data if new mount is empty
80+
DefaultDependencies=no
81+
After=var-lib-etcd.mount var.mount
82+
Before=crio.service
83+
84+
[Service]
85+
Type=oneshot
86+
RemainAfterExit=yes
87+
ExecCondition=/usr/bin/test ! -d /var/lib/etcd/member
88+
ExecStart=/usr/sbin/setenforce 0
89+
ExecStart=/bin/rsync -ar /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
90+
ExecStart=/usr/sbin/setenforce 1
91+
TimeoutSec=0
92+
93+
[Install]
94+
WantedBy=multi-user.target graphical.target
95+
enabled: true
96+
name: sync-var-lib-etcd-to-etcd.service
97+
- contents: |
98+
[Unit]
99+
Description=Restore recursive SELinux security contexts
100+
DefaultDependencies=no
101+
After=var-lib-etcd.mount
102+
Before=crio.service
103+
104+
[Service]
105+
Type=oneshot
106+
RemainAfterExit=yes
107+
ExecStart=/sbin/restorecon -R /var/lib/etcd/
108+
TimeoutSec=0
109+
110+
[Install]
111+
WantedBy=multi-user.target graphical.target
112+
enabled: true
113+
name: restorecon-var-lib-etcd.service
114+
115+
----
116+
117+
. Create the machine configuration by entering the following commands:
118+
+
119+
[source,terminal]
120+
----
121+
$ oc login -u ${ADMIN} -p ${ADMINPASSWORD} ${API}
122+
... output omitted ...
123+
----
124+
+
125+
[source,terminal]
126+
----
127+
$ oc create -f etcd-mc.yml
128+
machineconfig.machineconfiguration.openshift.io/98-var-lib-etcd created
129+
----
130+
+
131+
[source,terminal]
132+
----
133+
$ oc login -u ${ADMIN} -p ${ADMINPASSWORD} ${API}
134+
[... output omitted ...]
135+
----
136+
+
137+
[source, terminal]
138+
----
139+
$ oc create -f etcd-mc.yml machineconfig.machineconfiguration.openshift.io/98-var-lib-etcd created
140+
----
141+
+
142+
The nodes are updated and rebooted. After the reboot completes, the following events occur:
143+
+
144+
* An XFS file system is created on the specified disk.
145+
* The disk mounts to `/var/lib/etc`.
146+
* The content from `/sysroot/ostree/deploy/rhcos/var/lib/etcd` syncs to `/var/lib/etcd`.
147+
* A restore of `SELinux` labels is forced for `/var/lib/etcd`.
148+
* The old content is not removed.
149+
. After the nodes are on a separate disk, update the machine configuration file, `etcd-mc.yml` with the following information:
150+
+
151+
[source,yaml]
152+
----
153+
apiVersion: machineconfiguration.openshift.io/v1
154+
kind: MachineConfig
155+
metadata:
156+
labels:
157+
machineconfiguration.openshift.io/role: master
158+
name: 98-var-lib-etcd
159+
spec:
160+
config:
161+
ignition:
162+
version: 3.2.0
163+
systemd:
164+
units:
165+
- contents: |
166+
[Unit]
167+
Description=Mount /dev/sdb to /var/lib/etcd
168+
Before=local-fs.target
169+
[email protected]
170+
[email protected] var.mount
171+
172+
[Mount]
173+
What=/dev/sdb
174+
Where=/var/lib/etcd
175+
Type=xfs
176+
Options=defaults,prjquota
177+
178+
[Install]
179+
WantedBy=local-fs.target
180+
enabled: true
181+
name: var-lib-etcd.mount
182+
----
183+
. Apply the modified version that removes the logic for creating and syncing the device by entering the following command:
184+
+
185+
[source,terminal]
186+
----
187+
$ oc replace -f etcd-mc.yml
188+
----
189+
+
190+
The previous step prevents the nodes from rebooting.

‎scalability_and_performance/recommended-host-practices.adoc

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,12 @@ include::modules/recommended-etcd-practices.adoc[leveloffset=+1]
4545
.Additional resources
4646
* link:https://access.redhat.com/solutions/4885641[How to use `fio` to check etcd disk performance in {product-title}]
4747

48+
include::modules/move-etcd-different-disk.adoc[leveloffset=+1]
49+
50+
[role="_additional-resources"]
51+
.Additional resources
52+
* link:https://docs.openshift.com/container-platform/4.11/architecture/architecture-rhcos.html[Red Hat Enterprise Linux CoreOS (RHCOS)]
53+
4854
include::modules/etcd-defrag.adoc[leveloffset=+1]
4955

5056
include::modules/infrastructure-node-sizing.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)