Add procedure to configure kdump service with machine config

This adds a procedure for cluster-wide configuration of kdump service
with creating MachineConfig objects. Although the existing description
points the Installation configuration section, this section doesn't
provide enough information for enabling kdump such as how to enable a
systemd unit.

Author: Hironori Shiina

modules/investigating-kernel-crashes.adoc

@@ -27,7 +27,7 @@ Perform the following steps to enable `kdump` on {op-system}.
 # rpm-ostree kargs --append='crashkernel=256M'
 ----
 
-. Optional: To write the crash dump over the network or to some other location, rather than to the default local `/var/crash` location, edit the `/etc/kdump.conf` configuration file. 
+. Optional: To write the crash dump over the network or to some other location, rather than to the default local `/var/crash` location, edit the `/etc/kdump.conf` configuration file.
 +
 [NOTE]
 ====
@@ -65,6 +65,82 @@ The `kdump` service is intended to be enabled per node to debug kernel problems.
 
 If you are aware of the downsides and trade-offs of having the `kdump` service enabled, it is possible to enable `kdump` in a cluster-wide fashion. Although machine-specific machine configs are not yet supported, you can perform the previous steps through a `systemd` unit in a `MachineConfig` object on day-1 and have kdump enabled on all nodes in the cluster. You can create a `MachineConfig` object and inject that object into the set of manifest files used by Ignition during cluster setup. See "Customizing nodes" in the _Installing -> Installation configuration_ section for more information and examples on how to use Ignition configs.
 
+.Procedure
+
+Create a `MachineConfig` object for cluster-wide configuration:
+
+. Optional: If you change the `/etc/kdump.conf` configuration from the default, you can encode it into base64 format to include its content in your `MachineConfig` object:
++
+[source,terminal]
+----
+$ cat << EOF | base64
+path /var/crash
+core_collector makedumpfile -l --message-level 7 -d 31
+EOF
+----
+
+. Optional: Create a content of the `/etc/sysconfig/kdump` file and encode it as base64 if you change the configuration from the default:
++
+[source,terminal]
+----
+$ cat << EOF | base64
+KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet log_buf_len swiotlb"
+KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr novmcoredd hest_disable"
+KEXEC_ARGS="-s"
+KDUMP_IMG="vmlinuz"
+EOF
+----
+
+. Create the `MachineConfig` object file:
++
+[source,terminal]
+----
+$ cat << EOF > ./99-master-kdump-configuration.yaml
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: master <1>
+  name: 99-master-kdump-configuration
+spec:
+  kernelArguments:
+    - 'crashkernel=256M' <2>
+  config:
+    ignition:
+      version: 3.2.0
+    storage:
+      files:
+        - contents:
+            source: data:text/plain;charset=utf-8;base64,ICAgIHBhdGggL3Zhci9jcmFzaAogICAgY2...  <3>
+          mode: 420
+          overwrite: true
+          path: /etc/kdump.conf
+        - contents:
+            source: data:text/plain;charset=utf-8;base64,S0RVTVBfQ09NTUFORExJTkVfUkVNT1ZFPS...  <4>
+          mode: 420
+          overwrite: true
+          path: /etc/sysconfig/kdump
+    systemd:
+      units:
+        - enabled: true
+          name: kdump.service
+EOF
+----
++
+<1> Replace `master` with `worker` for creating a `MachineConfig` object for the `worker` role.
+<2> Provide kernel arguments to reserve memory for the crash kernel. You can add other kernel arguments if necessary.
+<3> Replace the base64 content with the one you created for `/etc/kdump.conf`.
+<4> Replace the base64 content with the one you created for `/etc/sysconfig/kdump`.
++
+
+. Put the YAML file into manifests during cluster setup. You can also create this `MachineConfig` object after cluster setup with the YAML file:
+
++
+[source,terminal]
+----
+ $ oc create -f ./99-master-kdump-configuration.yaml
+----
+
 == Testing the kdump configuration
 
 ifdef::openshift-enterprise[]