Merge pull request #53246 from jeana-redhat/OSDOCS-4235-CPMS-failure-recovery

jeana-redhat · web-flow · commit 4e767acdeb5a · 2022-11-30T14:05:54.000-05:00
[OSDOCS-4235]: CPMS resiliency and recovery
diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml
@@ -1977,6 +1977,8 @@ Topics:
     File: cpmso-configuration
   #- Name: Using the Control Plane Machine Set Operator
   #  File: cpmso-using
+  - Name: Control plane resiliency and recovery
+    File: cpmso-resiliency
   #- Name: Troubleshooting the Control Plane Machine Set Operator
   #  File: cpmso-troubleshooting
 - Name: Deploying machine health checks
diff --git a/machine_management/control_plane_machine_management/cpmso-resiliency.adoc b/machine_management/control_plane_machine_management/cpmso-resiliency.adoc
@@ -0,0 +1,35 @@
+:_content-type: ASSEMBLY
+[id="cpmso-resiliency"]
+= Control plane resiliency and recovery
+include::_attributes/common-attributes.adoc[]
+:context: cpmso-resiliency
+
+toc::[]
+
+You can use the Control Plane Machine Set Operator to improve the resiliency of the control plane for your {product-title} cluster. 
+
+[id="cpmso-failure-domains_{context}"]
+== High availability and fault tolerance with failure domains
+
+When possible, the control plane machine set spreads the control plane machines across multiple failure domains. This configuration provides high availability and fault tolerance within the control plane. This strategy can help protect the control plane when issues arise within the infrastructure provider.
+
+//Failure domain platform support and configuration
+include::modules/cpmso-failure-domains-provider.adoc[leveloffset=+2]
+
+[role="_additional-resources"]
+.Additional resources
+
+* xref:../../machine_management/control_plane_machine_management/cpmso-configuration.adoc#cpmso-yaml-failure-domain-aws_cpmso-configuration[Sample Amazon Web Services failure domain configuration]
+
+* xref:../../machine_management/control_plane_machine_management/cpmso-configuration.adoc#cpmso-yaml-failure-domain-azure_cpmso-configuration[Sample Microsoft Azure failure domain configuration]
+
+//Balancing control plane machines
+include::modules/cpmso-failure-domains-balancing.adoc[leveloffset=+2]
+
+//Recovery of the failed control plane machines
+include::modules/cpmso-control-plane-recovery.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+
+* xref:../../machine_management/deploying-machine-health-checks.adoc#deploying-machine-health-checks[Deploying machine health checks]
diff --git a/machine_management/control_plane_machine_management/cpmso-using.adoc b/machine_management/control_plane_machine_management/cpmso-using.adoc
@@ -11,9 +11,6 @@ The Control Plane Machine Set Operator automates the following capabilities:
 //Vertical resizing of the control plane
 include::modules/cpmso-feat-vertical-resize.adoc[leveloffset=+1]
 
-//Recovery of the failed control plane machines
-include::modules/cpmso-feat-failure-recovery.adoc[leveloffset=+1]
-
 //Updating the control plane configuration
 include::modules/cpmso-feat-config-update.adoc[leveloffset=+1]
 
diff --git a/machine_management/deploying-machine-health-checks.adoc b/machine_management/deploying-machine-health-checks.adoc
@@ -11,21 +11,14 @@ You can configure and deploy a machine health check to automatically repair dama
 include::modules/machine-user-provisioned-limitations.adoc[leveloffset=+1]
 
 include::modules/machine-health-checks-about.adoc[leveloffset=+1]
-
 [role="_additional-resources"]
 .Additional resources
-
-* For more information about the node conditions you can define in a `MachineHealthCheck` CR, see xref:../nodes/nodes/nodes-nodes-viewing.html#nodes-nodes-viewing-listing_nodes-nodes-viewing[About listing all the nodes in a cluster].
-
-* For more information about short-circuiting, see xref:../machine_management/deploying-machine-health-checks.adoc#machine-health-checks-short-circuiting_deploying-machine-health-checks[Short-circuiting machine health check remediation].
+* xref:../nodes/nodes/nodes-nodes-viewing.adoc#nodes-nodes-viewing-listing_nodes-nodes-viewing[About listing all the nodes in a cluster]
+* xref:../machine_management/deploying-machine-health-checks.adoc#machine-health-checks-short-circuiting_deploying-machine-health-checks[Short-circuiting machine health check remediation]
+* xref:../machine_management/control_plane_machine_management/cpmso-about.adoc#cpmso-about[About the Control Plane Machine Set Operator]
 
 include::modules/machine-health-checks-resource.adoc[leveloffset=+1]
 
-////
-[role="_additional-resources"]
-.Additional resources
-////
-
 include::modules/machine-health-checks-creating.adoc[leveloffset=+1]
 
 You can configure and deploy a machine health check to detect and repair unhealthy bare metal nodes.
diff --git a/modules/cpmso-control-plane-recovery.adoc b/modules/cpmso-control-plane-recovery.adoc
@@ -0,0 +1,20 @@
+// Module included in the following assemblies:
+//
+// * machine_management/cpmso-resiliency.adoc
+
+:_content-type: CONCEPT
+[id="cpmso-control-plane-recovery_{context}"]
+= Recovery of failed control plane machines
+
+The Control Plane Machine Set Operator automates the recovery of control plane machines. When a control plane machine is deleted, the Operator creates a replacement with the configuration that is specified in the `ControlPlaneMachineSet` custom resource (CR).  
+
+For clusters that use control plane machine sets, you can configure a machine health check. The machine health check deletes unhealthy control plane machines so that they are replaced. 
+
+[IMPORTANT]
+====
+If you configure a `MachineHealthCheck` resource for the control plane, set the value of `maxUnhealthy` to `1`. 
+
+This configuration ensures that the machine health check takes no action when multiple control plane machines appear to be unhealthy. Multiple unhealthy control plane machines can indicate that the etcd cluster is degraded or that a scaling operation to replace a failed machine is in progress. 
+
+If the etcd cluster is degraded, manual intervention might be required. If a scaling operation is in progress, the machine health check should allow it to finish.
+====
diff --git a/modules/cpmso-failure-domains-balancing.adoc b/modules/cpmso-failure-domains-balancing.adoc
@@ -0,0 +1,14 @@
+// Module included in the following assemblies:
+//
+// * machine_management/cpmso-resiliency.adoc
+
+:_content-type: CONCEPT
+[id="cpmso-failure-domains-balancing_{context}"]
+= Balancing control plane machines
+
+The control plane machine set balances control plane machines across the failure domains that are specified in the custom resource (CR).
+
+//If failure domains must be reused, they are selected alphabetically by name. 
+When possible, the control plane machine set uses each failure domain equally to ensure appropriate fault tolerance. If there are fewer failure domains than control plane machines, failure domains are selected for reuse alphabetically by name. For clusters with no failure domains specified, all control plane machines are placed within a single failure domain. 
+
+Some changes to the failure domain configuration cause the control plane machine set to rebalance the control plane machines. For example, if you add failure domains to a cluster with fewer failure domains than control plane machines, the control plane machine set rebalances the machines across all available failure domains. 
diff --git a/modules/cpmso-failure-domains-provider.adoc b/modules/cpmso-failure-domains-provider.adoc
@@ -0,0 +1,29 @@
+// Module included in the following assemblies:
+//
+// * machine_management/cpmso-resiliency.adoc
+
+:_content-type: REFERENCE
+[id="cpmso-failure-domains-provider_{context}"]
+= Failure domain platform support and configuration
+
+The control plane machine set concept of a failure domain is analogous to existing concepts on cloud providers. Not all platforms support the use of failure domains. 
+
+.Failure domain support matrix
+[cols="<.^,^.^,^.^"]
+|====
+|Cloud provider |Support for failure domains |Provider nomenclature
+
+|Amazon Web Services (AWS)
+|X
+|link:https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones[Availability Zone (AZ)]
+
+|Microsoft Azure
+|X
+|link:https://learn.microsoft.com/en-us/azure/azure-web-pubsub/concept-availability-zones[Azure availability zone]
+
+|VMware vSphere
+|
+|Not applicable
+|====
+
+The failure domain configuration in the control plane machine set custom resource (CR) is platform-specific. For more information about failure domain parameters in the CR, see the sample failure domain configuration for your provider.
diff --git a/modules/cpmso-feat-failure-recovery.adoc b/modules/cpmso-feat-failure-recovery.adoc
diff --git a/modules/cpmso-yaml-failure-domain-aws.adoc b/modules/cpmso-yaml-failure-domain-aws.adoc
@@ -6,7 +6,9 @@
 [id="cpmso-yaml-failure-domain-aws_{context}"]
 = Sample AWS failure domain configuration
 
-The `ControlPlaneMachineSet` CR spreads control plane machines across multiple failure domains when possible.
+The control plane machine set concept of a failure domain is analogous to existing AWS concept of an link:https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones[_Availability Zone (AZ)_]. The `ControlPlaneMachineSet` CR spreads control plane machines across multiple failure domains when possible. 
+
+When configuring AWS failure domains in the control plane machine set, you must specify the availability zone name and the subnet to use. 
 
 .Sample AWS failure domain values
 [source,yaml]
diff --git a/modules/cpmso-yaml-failure-domain-azure.adoc b/modules/cpmso-yaml-failure-domain-azure.adoc
@@ -6,7 +6,9 @@
 [id="cpmso-yaml-failure-domain-azure_{context}"]
 = Sample Azure failure domain configuration
 
-The `ControlPlaneMachineSet` CR spreads control plane machines across multiple failure domains when possible.
+The control plane machine set concept of a failure domain is analogous to existing Azure concept of an link:https://learn.microsoft.com/en-us/azure/azure-web-pubsub/concept-availability-zones[_Azure availability zone_]. The `ControlPlaneMachineSet` CR spreads control plane machines across multiple failure domains when possible. 
+
+When configuring Azure failure domains in the control plane machine set, you must specify the availability zone name.
 
 .Sample Azure failure domain values
 [source,yaml]
diff --git a/modules/machine-health-checks-about.adoc b/modules/machine-health-checks-about.adoc
@@ -9,13 +9,13 @@
 
 Machine health checks automatically repair unhealthy machines in a particular machine pool.
 
-To monitor machine health, create a resource to define the configuration for a controller. Set a condition to check, such as staying in the `NotReady` status for five minutes or displaying a permanent condition in the node-problem-detector, and a label for the set of machines to monitor.
-
 [NOTE]
 ====
-You cannot apply a machine health check to a machine with the master role.
+You can only apply a machine health check to control plane machines on clusters that use control plane machine sets.
 ====
 
+To monitor machine health, create a resource to define the configuration for a controller. Set a condition to check, such as staying in the `NotReady` status for five minutes or displaying a permanent condition in the node-problem-detector, and a label for the set of machines to monitor.
+
 The controller that observes a `MachineHealthCheck` resource checks for the defined condition. If a machine fails the health check, the machine is automatically deleted and one is created to take its place. When a machine is deleted, you see a `machine deleted` event.
 
 To limit disruptive impact of the machine deletion, the controller drains and deletes only one node at a time. If there are more unhealthy machines than the `maxUnhealthy` threshold allows for in the targeted pool of machines, remediation stops and therefore enables manual intervention.
@@ -36,7 +36,6 @@ To stop the check, remove the resource.
 There are limitations to consider before deploying a machine health check:
 
 * Only machines owned by a machine set are remediated by a machine health check.
-* Control plane machines are not currently supported and are not remediated if they are unhealthy.
 * If the node for a machine is removed from the cluster, a machine health check considers the machine to be unhealthy and remediates it immediately.
 * If the corresponding node for a machine does not join the cluster after the `nodeStartupTimeout`, the machine is remediated.
 * A machine is remediated immediately if the `Machine` resource phase is `Failed`.
diff --git a/modules/machine-health-checks-resource.adoc b/modules/machine-health-checks-resource.adoc
@@ -47,7 +47,7 @@ The `matchLabels` are examples only; you must map your machine groups based on y
 [id="machine-health-checks-short-circuiting_{context}"]
 == Short-circuiting machine health check remediation
 
-Short circuiting ensures that machine health checks remediate machines only when the cluster is healthy.
+Short-circuiting ensures that machine health checks remediate machines only when the cluster is healthy.
 Short-circuiting is configured through the `maxUnhealthy` field in the `MachineHealthCheck` resource.
 
 If the user defines a value for the `maxUnhealthy` field, before remediating any machines, the `MachineHealthCheck` compares the value of `maxUnhealthy` with the number of machines within its target pool that it has determined to be unhealthy. Remediation is not performed if the number of unhealthy machines exceeds the `maxUnhealthy` limit.
@@ -59,6 +59,15 @@ If `maxUnhealthy` is not set, the value defaults to `100%` and the machines are
 
 The appropriate `maxUnhealthy` value depends on the scale of the cluster you deploy and how many machines the `MachineHealthCheck` covers. For example, you can use the `maxUnhealthy` value to cover multiple compute machine sets across multiple availability zones so that if you lose an entire zone, your `maxUnhealthy` setting prevents further remediation within the cluster. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.
 
+[IMPORTANT]
+====
+If you configure a `MachineHealthCheck` resource for the control plane, set the value of `maxUnhealthy` to `1`. 
+
+This configuration ensures that the machine health check takes no action when multiple control plane machines appear to be unhealthy. Multiple unhealthy control plane machines can indicate that the etcd cluster is degraded or that a scaling operation to replace a failed machine is in progress. 
+
+If the etcd cluster is degraded, manual intervention might be required. If a scaling operation is in progress, the machine health check should allow it to finish.
+====
+
 The `maxUnhealthy` field can be set as either an integer or percentage.
 There are different remediation implementations depending on the `maxUnhealthy` value.
 
diff --git a/post_installation_configuration/node-tasks.adoc b/post_installation_configuration/node-tasks.adoc
@@ -63,6 +63,10 @@ Understand and deploy machine health checks.
 
 include::modules/machine-user-provisioned-limitations.adoc[leveloffset=+2]
 include::modules/machine-health-checks-about.adoc[leveloffset=+2]
+[role="_additional-resources"]
+.Additional resources
+* xref:../machine_management/control_plane_machine_management/cpmso-about.adoc#cpmso-about[About the Control Plane Machine Set Operator]
+
 include::modules/machine-health-checks-resource.adoc[leveloffset=+2]
 include::modules/machine-health-checks-creating.adoc[leveloffset=+2]
 include::modules/machineset-manually-scaling.adoc[leveloffset=+2]
diff --git a/scalability_and_performance/recommended-cluster-scaling-practices.adoc b/scalability_and_performance/recommended-cluster-scaling-practices.adoc
@@ -21,7 +21,10 @@ include::modules/recommended-scale-practices.adoc[leveloffset=+1]
 include::modules/machineset-modifying.adoc[leveloffset=+1]
 
 include::modules/machine-health-checks-about.adoc[leveloffset=+1]
+[role="_additional-resources"]
+.Additional resources
+* xref:../machine_management/control_plane_machine_management/cpmso-about.adoc#cpmso-about[About the Control Plane Machine Set Operator]
 
-include::modules/machine-health-checks-resource.adoc[leveloffset=+1]
+include::modules/machine-health-checks-resource.adoc[leveloffset=+2]
 
-include::modules/machine-health-checks-creating.adoc[leveloffset=+1]
+include::modules/machine-health-checks-creating.adoc[leveloffset=+2]
