Merge pull request openshift#7664 from shiftstack/rootvol-etcd-doc

OSASINFRA-3280: openstack: document etcd on local disk

Author: openshift-merge-bot[bot]

docs/user/openstack/customization.md

@@ -61,6 +61,8 @@ Beyond the [platform-agnostic `install-config.yaml` properties](../customization
 > **Note**
 > Note when deploying with `Kuryr` there is an Octavia API loadbalancer VM that will not fulfill the Availability Zones restrictions due to Octavia lack of support for it. In addition, if Octavia only has the amphora provider instead of also the OVN-Octavia provider, all the OpenShift services will be backed up by Octavia Load Balancer VMs which will not fulfill the Availability Zone restrictions either.
 
+> **Note**
+> Note when deploying the control-plane machines with `rootVolume`, it is highly suggested to use an [additional ephemeral disk dedicated to etcd](./etcd-ephemeral-disk.md).
 
 ## Examples
 

docs/user/openstack/etcd-ephemeral-disk.md

@@ -0,0 +1,230 @@
+# Moving etcd to an ephemeral local disk
+
+You can move etcd from a root volume (Cinder) to a dedicated ephemeral local disk to prevent or resolve performance issues.
+
+## Prerequisites
+
+* This migration is currently tested and documented as a day 2 operation.
+* An OpenStack cloud where Nova is configured to use local storage for ephemeral disks. The `libvirt.images_type` option in `nova.conf` must not be `rbd`.
+* An OpenStack cloud with Cinder being functional and enough available storage to accommodate 3 Root Volumes for the OpenShift control plane.
+* OpenShift will be deployed with IPI for now; UPI is not yet documented but technically possible.
+* The control-plane machine’s auxiliary storage device, such as /dev/vdb, must match the vdb. Change this reference in all places in the file.
+
+## Procedure
+
+* Create a Nova flavor for the Control Plane which allows 10 GiB of Ephemeral Disk:
+
+```bash
+openstack flavor create --ephemeral 10 [...]
+```
+
+* We will deploy a cluster with Root Volumes for the Control Plane. Here is an example of `install-config.yaml`:
+
+```yaml
+[...]
+controlPlane:
+  name: master
+  platform:
+    openstack:
+      type: ${CONTROL_PLANE_FLAVOR}
+      rootVolume:
+        size: 100
+        types:
+        - ${CINDER_TYPE}
+  replicas: 3
+[...]
+```
+
+* Run openshift-install with the following parameters to create the cluster:
+
+```bash
+openshift-install create cluster --dir=install_dir
+```
+
+* Once the cluster has been deployed and is healthy, edit the ControlPlaneMachineSet (CPMS) to add the additional block ephemeral device that will be used by etcd:
+
+```bash
+oc patch ControlPlaneMachineSet/cluster -n openshift-machine-api --type json -p '[{"op": "add", "path": "/spec/template/machines_v1beta1_machine_openshift_io/spec/providerSpec/value/additionalBlockDevices", "value": [{"name": "etcd", "sizeGiB": 10, "storage": {"type": "Local"}}]}]'
+```
+
+> [!NOTE]
+> Putting etcd on a block device of type Volume is not supported for performance reasons simply because we don't test it.
+> While it's functionally the same as using the root volume, we decided to support local devices only for now.
+
+* Wait for the control-plane to roll out with new Machines. A few commands can be used to check that everything is healthy:
+
+```bash
+oc wait --timeout=90m --for=condition=Progressing=false controlplanemachineset.machine.openshift.io -n openshift-machine-api cluster
+oc wait --timeout=90m --for=jsonpath='{.spec.replicas}'=3 controlplanemachineset.machine.openshift.io -n openshift-machine-api cluster
+oc wait --timeout=90m --for=jsonpath='{.status.updatedReplicas}'=3 controlplanemachineset.machine.openshift.io -n openshift-machine-api cluster
+oc wait --timeout=90m --for=jsonpath='{.status.replicas}'=3 controlplanemachineset.machine.openshift.io -n openshift-machine-api cluster
+oc wait --timeout=90m --for=jsonpath='{.status.readyReplicas}'=3 controlplanemachineset.machine.openshift.io -n openshift-machine-api cluster
+oc wait clusteroperators --timeout=30m --all --for=condition=Progressing=false
+```
+
+* Check that we have 3 control plane machines, and that each machine has the additional block device:
+
+```bash
+cp_machines=$(oc get machines -n openshift-machine-api --selector='machine.openshift.io/cluster-api-machine-role=master' --no-headers -o custom-columns=NAME:.metadata.name)
+if [[ $(echo "${cp_machines}" | wc -l) -ne 3 ]]; then
+  exit 1
+fi
+for machine in ${cp_machines}; do
+  if ! oc get machine -n openshift-machine-api "${machine}" -o jsonpath='{.spec.providerSpec.value.additionalBlockDevices}' | grep -q 'etcd'; then
+  exit 1
+  fi
+done
+```
+
+* We will use a MachineConfig to handle etcd on local disk. Create a file named `98-var-lib-etcd.yaml` with this content:
+
+```yaml
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: master
+  name: 98-var-lib-etcd
+spec:
+  config:
+    ignition:
+      version: 3.2.0
+    systemd:
+      units:
+      - contents: |
+          [Unit]
+          Description=Make File System on /dev/vdb
+          DefaultDependencies=no
+          BindsTo=dev-vdb.device
+          After=dev-vdb.device var.mount
+          [email protected]
+
+          [Service]
+          Type=oneshot
+          RemainAfterExit=yes
+          ExecStart=/usr/sbin/mkfs.xfs -f /dev/vdb
+          TimeoutSec=0
+
+          [Install]
+          WantedBy=var-lib-containers.mount
+        enabled: true
+        name: [email protected]
+      - contents: |
+          [Unit]
+          Description=Mount /dev/vdb to /var/lib/etcd
+          Before=local-fs.target
+          [email protected]
+          [email protected] var.mount
+
+          [Mount]
+          What=/dev/vdb
+          Where=/var/lib/etcd
+          Type=xfs
+          Options=defaults,prjquota
+
+          [Install]
+          WantedBy=local-fs.target
+        enabled: true
+        name: var-lib-etcd.mount
+      - contents: |
+          [Unit]
+          Description=Sync etcd data if new mount is empty
+          DefaultDependencies=no
+          After=var-lib-etcd.mount var.mount
+          Before=crio.service
+
+          [Service]
+          Type=oneshot
+          RemainAfterExit=yes
+          ExecCondition=/usr/bin/test ! -d /var/lib/etcd/member
+          ExecStart=/usr/sbin/setenforce 0
+          ExecStart=/bin/rsync -ar /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
+          ExecStart=/usr/sbin/setenforce 1
+          TimeoutSec=0
+
+          [Install]
+          WantedBy=multi-user.target graphical.target
+        enabled: true
+        name: sync-var-lib-etcd-to-etcd.service
+      - contents: |
+          [Unit]
+          Description=Restore recursive SELinux security contexts
+          DefaultDependencies=no
+          After=var-lib-etcd.mount
+          Before=crio.service
+
+          [Service]
+          Type=oneshot
+          RemainAfterExit=yes
+          ExecStart=/sbin/restorecon -R /var/lib/etcd/
+          TimeoutSec=0
+
+          [Install]
+          WantedBy=multi-user.target graphical.target
+        enabled: true
+        name: restorecon-var-lib-etcd.service
+```
+
+* Apply this file that will create the device and sync the data by entering the following command:
+
+```bash
+oc create -f 98-var-lib-etcd.yaml
+```
+
+* This will take some time to complete, as the etcd data will be synced from the root volume to the local disk on
+the control-plane machines. Run these commands to check whether the cluster is healthy:
+
+```bash
+oc wait --timeout=45m --for=condition=Updating=false machineconfigpool/master
+oc wait node --selector='node-role.kubernetes.io/master' --for condition=Ready --timeout=30s
+oc wait clusteroperators --timeout=30m --all --for=condition=Progressing=false
+```
+
+
+* Once the cluster is healthy, create a file named `etcd-replace.yaml` with this content:
+
+```yaml
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: master
+  name: 98-var-lib-etcd
+spec:
+  config:
+    ignition:
+      version: 3.2.0
+    systemd:
+      units:
+      - contents: |
+          [Unit]
+          Description=Mount /dev/vdb to /var/lib/etcd
+          Before=local-fs.target
+          [email protected]
+          [email protected] var.mount
+
+          [Mount]
+          What=/dev/vdb
+          Where=/var/lib/etcd
+          Type=xfs
+          Options=defaults,prjquota
+
+          [Install]
+          WantedBy=local-fs.target
+        enabled: true
+        name: var-lib-etcd.mount
+```
+
+Apply this file that will remove the logic for creating and syncing the device by entering the following command:
+
+```bash
+oc replace -f etcd-replace.yaml
+```
+
+* Again we need to wait for the cluster to be healthy. The same commands as above can be used to check that everything is healthy.
+
+* Now etcd is stored on ephemeral local disk. This can be verified by connected to a master nodes with `oc debug node/<master-node-name>` and running the following commands:
+
+```bash
+oc debug node/<master-node-name> -- df -T /host/var/lib/etcd
+```