OSDOCS-3040: Check for config drift regularly

mburke5678 · mburke5678 · commit d847cf7bc139 · 2022-01-05T15:27:24.000-05:00
diff --git a/architecture/control-plane.adoc b/architecture/control-plane.adoc
@@ -9,6 +9,10 @@ include::modules/understanding-control-plane.adoc[leveloffset=+1]
 
 include::modules/architecture-machine-config-pools.adoc[leveloffset=+2]
 
+.Additional resources
+
+* For more information on configuration drift detection, see xref:../post_installation_configuration/machine-configuration-tasks.adoc#machine-config-drift-detection_post-install-machine-configuration-tasks[Understanding configuration drift detection].
+
 include::modules/architecture-machine-roles.adoc[leveloffset=+2]
 
 include::modules/operators-overview.adoc[leveloffset=+2]
@@ -19,4 +23,6 @@ include::modules/understanding-machine-config-operator.adoc[leveloffset=+3]
 
 .Additional information
 
-For information on preventing the control plane machines from rebooting after the Machine Config Operator makes changes to the machine config, see xref:../support/troubleshooting/troubleshooting-operator-issues.adoc#troubleshooting-disabling-autoreboot-mco_troubleshooting-operator-issues[Disabling Machine Config Operator from automatically rebooting].
+* For more information on detecting configuration drift, see xref:../post_installation_configuration/machine-configuration-tasks.adoc#machine-config-drift-detection_post-install-machine-configuration-tasks[Understanding configuration drift detection].
+
+* For information on preventing the control plane machines from rebooting after the Machine Config Operator makes changes to the machine config, see xref:../support/troubleshooting/troubleshooting-operator-issues.adoc#troubleshooting-disabling-autoreboot-mco_troubleshooting-operator-issues[Disabling Machine Config Operator from automatically rebooting].
diff --git a/modules/architecture-machine-config-pools.adoc b/modules/architecture-machine-config-pools.adoc
@@ -24,3 +24,6 @@ Any node labeled with the `infra` role that is only running infra workloads is n
 ====
 
 The MCO applies updates for pools independently; for example, if there is an update that affects all pools, nodes from each pool update in parallel with each other. If you add a custom pool, nodes from that pool also attempt to update concurrently with the master and worker nodes.
+
+There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called _configuration drift_. The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node `degraded` until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated.
+
diff --git a/modules/machine-config-drift-detection.adoc b/modules/machine-config-drift-detection.adoc
@@ -0,0 +1,105 @@
+// Module included in the following assemblies:
+//
+// * post_installation_configuration/machine-configuration-tasks.adoc
+
+[id="machine-config-drift-detection_{context}"]
+= Understanding configuration drift detection
+
+There might be situations when the on-disk state of a node differs from what is configured in the machine config. This is known as _configuration drift_. For example, a cluster admin might manually modify a file, a systemd unit file, or a file permission that was configured through a machine config. This causes configuration drift. Configuration drift can cause problems between nodes in a Machine Config Pool or when the machine configs are updated.  
+
+The Machine Config Operator (MCO) uses the Machine Config Daemon (MCD) to check nodes for configuration drift on a regular basis. If detected, the MCO sets the node and the machine config pool (MCP) to `Degraded` and reports the error. A degraded node is online and operational, but, it cannot be updated.
+
+The MCD performs configuration drift detection upon each of the following conditions:
+
+* When a node boots.
+* After any of the files (Ignition files and systemd drop-in units) specified in the machine config are modified outside of the machine config.
+* Before a new machine config is applied.
++
+[NOTE]
+====
+If you apply a new machine config to the nodes, the MCD temporarily shuts down configuration drift detection. This shutdown is needed because the new machine config necessarily differs from the machine config on the nodes. After the new machine config is applied, the MCD restarts detecting configuration drift using the new machine config.
+====
+
+When performing configuration drift detection, the MCD validates that the file contents and permissions fully match what the currently-applied machine config specifies. Typically, the MCD detects configuration drift in less than a second after the detection is triggered. 
+
+If the MCD detects configuration drift, the MCD performs the following tasks:
+
+* Emits an error to the console logs
+* Emits a Kubernetes event
+* Stops further detection on the node
+* Sets the node and MCP to `degraded`
+
+You can check if you have a degraded node by listing the MCPs:
+
+[source,terminal]
+----
+$ oc get mcp worker
+----
+
+If you have a degraded MCP, the `DEGRADEDMACHINECOUNT` field is non-zero, similar to the following output:
+
+.Example output
+[source,terminal]
+----
+NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
+worker   rendered-worker-404caf3180818d8ac1f50c32f14b57c3   False     True       True       2              1                   1                     1                      5h51m
+----
+
+You can determine if the problem is caused by configuration drift by examining the machine config pool:
+
+[source,terminal]
+----
+$ oc describe mcp worker
+----
+
+.Example output
+[source,terminal]
+----
+ ...
+    Last Transition Time:  2021-12-20T18:54:00Z
+    Message:               Node ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4 is reporting: "content mismatch for file \"/etc/mco-test-file\"" <1>
+    Reason:                1 nodes are reporting degraded status on sync
+    Status:                True
+    Type:                  NodeDegraded <2>
+ ...
+----
+<1> This message shows that a node's `/etc/mco-test-file` file, which was added by the machine config, has changed outside of the machine config.
+<2> The state of the node is `NodeDegraded`.
+
+Or, if you know which node is degraded, examine that node:
+
+[source,terminal]
+----
+$ oc describe node/ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4
+----
+
+.Example output
+[source,terminal]
+----
+ ...
+
+Annotations:        cloud.network.openshift.io/egress-ipconfig: [{"interface":"nic0","ifaddr":{"ipv4":"10.0.128.0/17"},"capacity":{"ip":10}}]
+                    csi.volume.kubernetes.io/nodeid:
+                      {"pd.csi.storage.gke.io":"projects/openshift-gce-devel-ci/zones/us-central1-a/instances/ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4"}
+                    machine.openshift.io/machine: openshift-machine-api/ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4
+                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
+                    machineconfiguration.openshift.io/currentConfig: rendered-worker-67bd55d0b02b0f659aef33680693a9f9
+                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-67bd55d0b02b0f659aef33680693a9f9
+                    machineconfiguration.openshift.io/reason: content mismatch for file "/etc/mco-test-file" <1>
+                    machineconfiguration.openshift.io/state: Degraded <2>
+ ...
+----
+<1> The error message indicating that configuration drift was detected between the node and the listed machine config. Here the error message indicates that the contents of the `/etc/mco-test-file`, which was added by the machine config, has changed outside of the machine config.
+<2> The state of the node is `Degraded`.
+
+You can correct configuration drift and return the node to the `Ready` state by performing one of the following remediations:
+
+* Ensure that the contents and file permissions of the files on the node match what is configured in the machine config. You can manually rewrite the file
+contents or change the file permissions.
+* Generate a link:https://access.redhat.com/solutions/5414371[force file] on the degraded node. The force file causes the MCD to bypass the usual configuration drift detection and reapplies the current machine config.
++
+[NOTE]
+====
+Generating a force file on a node causes that node to reboot. 
+====
+
diff --git a/modules/machine-config-overview.adoc b/modules/machine-config-overview.adoc
@@ -66,6 +66,8 @@ The MCO is not the only Operator that can change operating system components on
 
 Tasks for the MCO configuration that can be done post-installation are included in the following procedures. See descriptions of {op-system} bare metal installation for system configuration tasks that must be done during or before {product-title} installation.
 
+There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called _configuration drift_. The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node `degraded` until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated. For more information on configuration drift, see _Understanding configuration drift detection_.     
+
 == Project
 
 See the link:https://github.com/openshift/machine-config-operator[openshift-machine-config-operator] GitHub site for details.
diff --git a/modules/understanding-machine-config-operator.adoc b/modules/understanding-machine-config-operator.adoc
@@ -55,3 +55,5 @@ The following modifications do not trigger a node reboot:
 
 * When the MCO detects changes to the `/etc/containers/registries.conf` file, such as adding or editing an `ImageContentSourcePolicy` object, it drains the corresponding nodes, applies the changes, and uncordons the nodes.
 ====
+
+There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called _configuration drift_. The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node `degraded` until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated.
diff --git a/post_installation_configuration/machine-configuration-tasks.adoc b/post_installation_configuration/machine-configuration-tasks.adoc
@@ -16,6 +16,7 @@ Tasks in this section describe how to use features of the Machine Config Operato
 
 include::modules/machine-config-operator.adoc[leveloffset=+2]
 include::modules/machine-config-overview.adoc[leveloffset=+2]
+include::modules/machine-config-drift-detection.adoc[leveloffset=+2]
 include::modules/checking-mco-status.adoc[leveloffset=+2]
 
 [id="using-machineconfigs-to-change-machines"]
@@ -26,6 +27,8 @@ link:https://access.redhat.com/solutions/4510281[updating] SSH authorized keys,
 
 {product-title} supports link:https://coreos.github.io/ignition/configuration-v3_2/[Ignition specification version 3.2]. All new machine configs you create going forward should be based on Ignition specification version 3.2. If you are upgrading your {product-title} cluster, any existing Ignition specification version 2.x machine configs will be translated automatically to specification version 3.2.
 
+There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called _configuration drift_. The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node `degraded` until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated. For more information on configuration drift, see xref:../post_installation_configuration/machine-configuration-tasks.adoc#machine-config-drift-detection_post-install-machine-configuration-tasks[Understanding configuration drift detection].
+
 include::modules/installation-special-config-chrony.adoc[leveloffset=+2]
 include::modules/cnf-disable-chronyd.adoc[leveloffset=+2]
 include::modules/nodes-nodes-kernel-arguments.adoc[leveloffset=+2]
