Merge pull request #45313 from mburke5678/MCO-77-pause-mcp

OSDOCS-3039: Improve kubelet CA update for Paused Machine Config Pools

Author: mburke5678

modules/compliance-applying.adoc

@@ -16,6 +16,11 @@ After the Compliance Operator processes the applied remediation, the `status.App
 
 Note that when the Machine Config Operator applies a new `MachineConfig` object to nodes in a pool, all the nodes belonging to the pool are rebooted. This might be inconvenient when applying multiple remediations, each of which re-renders the composite `75-$scan-name-$suite-name` `MachineConfig` object. To prevent applying the remediation immediately, you can pause the machine config pool by setting the `.spec.paused` attribute of a `MachineConfigPool` object to `true`.
 
+[NOTE]
+====
+Make sure the pools are unpaused when the CA certificate rotation happens. If the MCPs are paused, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated.
+====
+
 The Compliance Operator can apply remediations automatically. Set `autoApplyRemediations: true` in the `ScanSetting` top-level object.
 
 [WARNING]

modules/troubleshooting-disabling-autoreboot-mco.adoc

@@ -24,7 +24,11 @@ To avoid unwanted disruptions, you can modify the machine config pool (MCP) to p
 
 [NOTE]
 ====
-Pausing an MCP prevents the MCO from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically-rotated certificates from being pushed to the associated nodes, including the automatic rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires, and the MCO attempts to renew the certificate automatically, the new certificate is created but not applied across the nodes in the paused MCP. This causes failure in multiple `oc` commands, including but not limited to `oc debug`, `oc logs`, `oc exec`, and `oc attach`. Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only.
+Pausing an MCP prevents the MCO from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically rotated certificates from being pushed to the associated nodes, including the automatic rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. 
+
+If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires, and the MCO attempts to renew the certificate automatically, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated.
+
+Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only. 
 
 New CA certificates are generated at 292 days from the installation date and removed at 365 days from that date. To determine the next automatic CA certificate rotation, see the link:https://access.redhat.com/articles/5651701[Understand CA cert auto renewal in Red Hat OpenShift 4].
 ====

modules/understanding-machine-config-operator.adoc

@@ -46,6 +46,8 @@ When changes are made to a machine configuration, the Machine Config Operator (M
 
 To prevent the nodes from automatically rebooting after machine configuration changes, before making the changes, you must pause the autoreboot process by setting the `spec.paused` field to `true` in the corresponding machine config pool. When paused, machine configuration changes are not applied until you set the `spec.paused` field to `false` and the nodes have rebooted into the new configuration.
 
+Make sure the pools are unpaused when the CA certificate rotation happens. If the MCPs are paused, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated.
+
 The following modifications do not trigger a node reboot:
 
 * When the MCO detects any of the following changes, it applies the update without draining or rebooting the node:

modules/update-using-custom-machine-config-pools-about.adoc

@@ -26,7 +26,9 @@ Do not remove the default worker label from the nodes. The nodes *must* have a r
 +
 [NOTE]
 ====
-Pausing the MCP also pauses the kube-apiserver-to-kubelet-signer automatic CA certificates rotation. New CA certificates are generated at 292 days from the installation date and old certificates are removed 365 days from the installation date. See the link:https://access.redhat.com/articles/5651701[Understand CA cert auto renewal in Red Hat OpenShift 4] to find out how much time you have before the next automatic CA certificate rotation. Make sure the pools are unpaused when the CA cert rotation happens. If the MCPs are paused, the cert rotation does not happen, which causes the cluster to become degraded and causes failure in multiple `oc` commands, including but not limited to `oc debug`, `oc logs`, `oc exec`, and `oc attach`.
+Pausing the MCP also pauses the `kube-apiserver-to-kubelet-signer` automatic CA certificates rotation. New CA certificates are generated at 292 days from the installation date and old certificates are removed 365 days from the installation date. See the link:https://access.redhat.com/articles/5651701[Understand CA cert auto renewal in Red Hat OpenShift 4] to find out how much time you have before the next automatic CA certificate rotation. 
+
+Make sure the pools are unpaused when the CA certificate rotation happens. If the MCPs are paused, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated.
 ====
 
 . Perform the cluster update. The update process updates the MCPs that are not paused, including the control plane nodes.

modules/update-using-custom-machine-config-pools-canary.adoc

@@ -26,7 +26,11 @@ The rolling update process described in this topic involves:
 
 [NOTE]
 ====
-Pausing an MCP prevents the Machine Config Operator from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically-rotated certificates from being pushed to the associated nodes, including the automatic CA rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires and the MCO attempts to automatically renew the certificate, the new certificate is created but not applied across the nodes in the respective machine config pool. This causes failure in multiple `oc` commands, including but not limited to `oc debug`, `oc logs`, `oc exec`, and `oc attach`. Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only.
+Pausing an MCP prevents the Machine Config Operator from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically rotated certificates from being pushed to the associated nodes, including the automatic CA rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. 
+
+If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires and the MCO attempts to automatically renew the certificate, the new certificate is created but not applied across the nodes in the respective machine config pool. This causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated.
+
+Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only.
 ====
 
 //link that follows is in the assembly: updating-cluster-between-minor

modules/update-using-custom-machine-config-pools-pause.adoc

@@ -9,7 +9,9 @@ In this canary rollout update process, after you label the nodes that you do not
 
 [NOTE]
 ====
-Pausing the MCP also pauses the kube-apiserver-to-kubelet-signer automatic CA certificates rotation. New CA certificates are generated at 292 days from the installation date and old certificates are removed 365 days from the installation date. See the link:https://access.redhat.com/articles/5651701[Understand CA cert auto renewal in Red Hat OpenShift 4] to find out how much time you have before the next automatic CA certificate rotation. Make sure the pools are unpaused when the CA cert rotation happens. If the MCPs are paused, the cert rotation does not happen, which causes the cluster to become degraded and causes failure in multiple `oc` commands, including but not limited to `oc debug`, `oc logs`, `oc exec`, and `oc attach`.
+Pausing the MCP also pauses the `kube-apiserver-to-kubelet-signer` automatic CA certificates rotation. New CA certificates are generated at 292 days from the installation date and old certificates are removed 365 days from the installation date. See the link:https://access.redhat.com/articles/5651701[Understand CA cert auto renewal in Red Hat OpenShift 4] to find out how much time you have before the next automatic CA certificate rotation. 
+
+Make sure the pools are unpaused when the CA certificate rotation happens. If the MCPs are paused, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated.
 ====
 
 To pause an MCP:

updating/update-using-custom-machine-config-pools.adoc

@@ -28,7 +28,11 @@ This scenario has not been tested and might result in an undefined cluster state
 
 [IMPORTANT]
 ====
-Pausing a machine config pool prevents the Machine Config Operator from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically-rotated certificates from being pushed to the associated nodes, including the automatic CA rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires and the MCO attempts to automatically renew the certificate, the new certificate is created but not applied across the nodes in the respective machine config pool. This causes failure in multiple `oc` commands, including but not limited to `oc debug`, `oc logs`, `oc exec`, and `oc attach`. Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only.
+Pausing a machine config pool prevents the Machine Config Operator from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically rotated certificates from being pushed to the associated nodes, including the automatic CA rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. 
+
+If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires and the MCO attempts to automatically renew the certificate, the MCO cannot push the newly rotated certificates to those nodes. This causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated.
+
+Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only.
 ====
 
 [id="update-using-custom-machine-config-pools-about-mcp_{context}"]