Skip to content

Commit 391c822

Browse files
mburke5678mburke5678
authored
Merge pull request #40644 from mburke5678/OSDOCS-3167-node-separate-var-partition
OSDOCS-3167: Test, Document and Support adding a node with a separate /var partition
2 parents 36e5831 + 997f669 commit 391c822

File tree

2 files changed

+300
-0
lines changed

2 files changed

+300
-0
lines changed

‎modules/machine-node-custom-partition.adoc‎

Lines changed: 294 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,294 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * machine_management/
4+
// * machine_management/
5+
// * post_installation_configuration/node-tasks.adoc
6+
7+
:_content-type: PROCEDURE
8+
[id="machine-node-custom-partition_{context}"]
9+
= Adding a new {op-system} worker node with a custom `/var` partition in AWS
10+
11+
{product-title} supports partitioning devices during installation by using machine configs that are processed during the bootstrap. However, if you use `/var` partitioning, the device name must be determined at installation and cannot be changed. You cannot add different instance types as nodes if they have a different device naming schema. For example, if you configured the `/var` partition with the default AWS device name for `m4.large` instances, `dev/xvdb`, you cannot directly add an AWS `m5.large` instance, as `m5.large` instances use a `/dev/nvme1n1` device by default. The device might fail to partition due to the different naming schema.
12+
13+
The procedure in this section shows how to add a new {op-system-first} compute node with an instance that uses a different device name from what was configured at installation. You create a custom user data secret and configure a new machine set. These steps are specific to an AWS cluster. The principles apply to other cloud deployments also. However, the device naming schema is different for other deployments and should be determined on a per-case basis.
14+
15+
.Procedure
16+
17+
. On a command line, change to the `openshift-machine-api` namespace:
18+
+
19+
[source,terminal]
20+
----
21+
$ oc project openshift-machine-api
22+
----
23+
24+
. Create a new secret from the `worker-user-data` secret:
25+
26+
.. Export the `userData` section of the secret to a text file:
27+
+
28+
[source,terminal]
29+
----
30+
$ oc get secret worker-user-data --template='{{index .data.userData | base64decode}}' | jq > userData.txt
31+
----
32+
33+
.. Edit the text file to add the `storage`, `filesystems`, and `systemd` stanzas for the partitions you want to use for the new node. You can specify any link:https://coreos.github.io/ignition/configuration-v3_2/[Ignition configuration parameters] as needed.
34+
+
35+
[NOTE]
36+
====
37+
Do not change the values in the `ignition` stanza.
38+
====
39+
+
40+
[source,terminal]
41+
----
42+
{
43+
"ignition": {
44+
"config": {
45+
"merge": [
46+
{
47+
"source": "https:...."
48+
}
49+
]
50+
},
51+
"security": {
52+
"tls": {
53+
"certificateAuthorities": [
54+
{
55+
"source": "data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lJTjRGN3RORnlWMGd3RFFZSktvWklodmNOQVFFTEJRQXdKakVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1SQXdEZ1lEVlFRREV3ZHliMjkwTFdOaE1CNFhEVEl5TURFeE56RTVOREF4T1ZvWApEVE15TURFeE5URTVOREF4T1Zvd0pqRVNNQkFHQTFVRUN4TUpiM0JsYm5Ob2FXWjBNUkF3RGdZRFZRUURFd2R5CmIyOTBMV05oTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF3OFBRODR2SkM4WVkKL3lQK3lSeHpoMWdKa2RHcmJVRVNNzUnh5c0l6YW0rczFISGVtcS90MWN3SEh1NCtrRTNVVllCanlpdE9=="
56+
}
57+
]
58+
}
59+
},
60+
"version": "3.2.0"
61+
}
62+
}
63+
64+
}
65+
]
66+
},
67+
"security": {
68+
"tls": {
69+
"certificateAuthorities": [
70+
{
71+
"source": "data:text/plain;charset=utf-8;base64,.....=="
72+
}
73+
]
74+
}
75+
},
76+
"version": "3.2.0"
77+
},
78+
"storage": {
79+
"disks": [
80+
{
81+
"device": "/dev/nvme1n1", <1>
82+
"partitions": [
83+
{
84+
"label": "var",
85+
"sizeMiB": 50000 <2>
86+
"startMiB": 0 <3>
87+
}
88+
]
89+
}
90+
],
91+
"filesystems": [
92+
{
93+
"device": "/dev/disk/by-partlabel/var", <4>
94+
"format": "xfs", <5>
95+
"path": "/var" <6>
96+
}
97+
]
98+
},
99+
"systemd": {
100+
"units": [ <7>
101+
{
102+
"contents": "[Unit]\nBefore=local-fs.target\n[Mount]\nWhere=/var\nWhat=/dev/disk/by-partlabel/var\nOptions=defaults,pquota\n[Install]\nWantedBy=local-fs.target\n",
103+
"enabled": true,
104+
"name": "var.mount"
105+
}
106+
]
107+
}
108+
}
109+
----
110+
//Copied from installation-disk-partitioning-upi-templates.adoc
111+
<1> Specifies an absolute path to the AWS block device.
112+
<2> Specifies the size of the data partition in Mebibytes.
113+
<3> Specifies the start of the partition in Mebibytes. When adding a data partition to the boot disk, a minimum value of 25000 MB (Mebibytes) is recommended. The root file system is automatically resized to fill all available space up to the specified offset. If no value is specified, or if the specified value is smaller than the recommended minimum, the resulting root file system will be too small, and future reinstalls of {op-system} might overwrite the beginning of the data partition.
114+
<4> Specifies an absolute path to the `/var` partition.
115+
<5> Specifies the filesystem format.
116+
<6> Specifies the mount-point of the filesystem while Ignition is running relative to where the root filesystem will be mounted. This is not necessarily the same as where it should be mounted in the real root, but it is encouraged to make it the same.
117+
<7> Defines a systemd mount unit that mounts the `/dev/disk/by-partlabel/var` device to the `/var` partition.
118+
119+
.. Extract the `disableTemplating` section from the `work-user-data` secret to a text file:
120+
+
121+
[source,terminal]
122+
----
123+
$ oc get secret worker-user-data --template='{{index .data.disableTemplating | base64decode}}' | jq > disableTemplating.txt
124+
----
125+
126+
.. Create the new user data secret file from the two text files. This user data secret passes the additional node partition information in the `userData.txt` file to the newly created node.
127+
+
128+
[source,terminal]
129+
----
130+
$ oc create secret generic worker-user-data-x5 --from-file=userData=userData.txt --from-file=disableTemplating=disableTemplating.txt
131+
----
132+
133+
. Create a new machine set for the new node:
134+
135+
.. Create a new machine set YAML file, similar to the following, which is configured for AWS. Add the required partitions and the newly-created user data secret:
136+
+
137+
[TIP]
138+
====
139+
Use an existing machine set as a template and change the parameters as needed for the new node.
140+
====
141+
+
142+
[source,terminal]
143+
----
144+
apiVersion: machine.openshift.io/v1beta1
145+
kind: MachineSet
146+
metadata:
147+
labels:
148+
machine.openshift.io/cluster-api-cluster: auto-52-92tf4
149+
name: worker-us-east-2-nvme1n1 <1>
150+
namespace: openshift-machine-api
151+
spec:
152+
replicas: 1
153+
selector:
154+
matchLabels:
155+
machine.openshift.io/cluster-api-cluster: auto-52-92tf4
156+
machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
157+
template:
158+
metadata:
159+
labels:
160+
machine.openshift.io/cluster-api-cluster: auto-52-92tf4
161+
machine.openshift.io/cluster-api-machine-role: worker
162+
machine.openshift.io/cluster-api-machine-type: worker
163+
machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
164+
spec:
165+
metadata: {}
166+
providerSpec:
167+
value:
168+
ami:
169+
id: ami-0c2dbd95931a
170+
apiVersion: awsproviderconfig.openshift.io/v1beta1
171+
blockDevices:
172+
- DeviceName: /dev/nvme1n1 <2>
173+
ebs:
174+
encrypted: true
175+
iops: 0
176+
volumeSize: 120
177+
volumeType: gp2
178+
- DeviceName: /dev/nvme1n2 <3>
179+
ebs:
180+
encrypted: true
181+
iops: 0
182+
volumeSize: 50
183+
volumeType: gp2
184+
credentialsSecret:
185+
name: aws-cloud-credentials
186+
deviceIndex: 0
187+
iamInstanceProfile:
188+
id: auto-52-92tf4-worker-profile
189+
instanceType: m5.large
190+
kind: AWSMachineProviderConfig
191+
metadata:
192+
creationTimestamp: null
193+
placement:
194+
availabilityZone: us-east-2b
195+
region: us-east-2
196+
securityGroups:
197+
- filters:
198+
- name: tag:Name
199+
values:
200+
- auto-52-92tf4-worker-sg
201+
subnet:
202+
id: subnet-07a90e5db1
203+
tags:
204+
- name: kubernetes.io/cluster/auto-52-92tf4
205+
value: owned
206+
userDataSecret:
207+
name: worker-user-data-x5 <4>
208+
----
209+
<1> Specifies a name for the new node.
210+
<2> Specifies an absolute path to the AWS block device, here an encrypted EBS volume.
211+
<3> Optional. Specifies an additional EBS volume.
212+
<4> Specifies the user data secret file.
213+
214+
.. Create the machine set:
215+
+
216+
[source,yaml]
217+
----
218+
$ oc create -f <file-name>.yaml
219+
----
220+
+
221+
The machines might take a few moments to become available.
222+
223+
. Verify that the new partition and nodes are created:
224+
225+
.. Verify that the machine set is created:
226+
+
227+
[source,terminal]
228+
----
229+
$ oc get machineset
230+
----
231+
+
232+
.Example output
233+
+
234+
[source,terminal]
235+
----
236+
NAME DESIRED CURRENT READY AVAILABLE AGE
237+
ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1a 1 1 1 1 124m
238+
ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1b 2 2 2 2 124m
239+
worker-us-east-2-nvme1n1 1 1 1 1 2m35s <1>
240+
----
241+
<1> This is the new machine set.
242+
243+
.. Verify that the new node is created:
244+
+
245+
[source,terminal]
246+
----
247+
$ oc get nodes
248+
----
249+
+
250+
.Example output
251+
+
252+
[source,terminal]
253+
----
254+
NAME STATUS ROLES AGE VERSION
255+
ip-10-0-128-78.ec2.internal Ready worker 117m v1.23.0+60f5a1c
256+
ip-10-0-146-113.ec2.internal Ready master 127m v1.23.0+60f5a1c
257+
ip-10-0-153-35.ec2.internal Ready worker 118m v1.23.0+60f5a1c
258+
ip-10-0-176-58.ec2.internal Ready master 126m v1.23.0+60f5a1c
259+
ip-10-0-217-135.ec2.internal Ready worker 2m57s v1.23.0+60f5a1c <1>
260+
ip-10-0-225-248.ec2.internal Ready master 127m v1.23.0+60f5a1c
261+
ip-10-0-245-59.ec2.internal Ready worker 116m v1.23.0+60f5a1c
262+
----
263+
<1> This is new new node.
264+
265+
.. Verify that the custom `/var` partition is created on the new node:
266+
+
267+
[source,terminal]
268+
----
269+
$ oc debug node/<node-name> -- chroot /host lsblk
270+
----
271+
+
272+
For example:
273+
+
274+
[source,terminal]
275+
----
276+
$ oc debug node/ip-10-0-217-135.ec2.internal -- chroot /host lsblk
277+
----
278+
+
279+
.Example output
280+
+
281+
[source,terminal]
282+
----
283+
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
284+
nvme0n1 202:0 0 120G 0 disk
285+
|-nvme0n1p1 202:1 0 1M 0 part
286+
|-nvme0n1p2 202:2 0 127M 0 part
287+
|-nvme0n1p3 202:3 0 384M 0 part /boot
288+
`-nvme0n1p4 202:4 0 119.5G 0 part /sysroot
289+
nvme1n1 202:16 0 50G 0 disk
290+
`-nvme1n1p1 202:17 0 48.8G 0 part /var <1>
291+
----
292+
<1> The `nvme1n1` device is mounted to the `/var` partition.
293+
294+

‎post_installation_configuration/node-tasks.adoc‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,12 @@ include::modules/machine-user-infra-machines-pxe.adoc[leveloffset=+2]
4848

4949
include::modules/installation-approve-csrs.adoc[leveloffset=+2]
5050

51+
include::modules/machine-node-custom-partition.adoc[leveloffset=+2]
52+
53+
.Additional resources
54+
55+
* For more information on how {product-title} uses disk partitioning, see xref:../installing/installing_bare_metal/installing-bare-metal.adoc#installation-user-infra-machines-advanced_disk_installing-bare-metal[Disk partitioning].
56+
5157
[id="post-installation-config-deploying-machine-health-checks"]
5258
== Deploying machine health checks
5359

0 commit comments

Comments
 (0)