Merge pull request #57986 from ogradyp/TELCODOCS-329

TELCODOCS-329: MachineHealthCheck: Implement metal3 remediation using new external API

Author: kelbrown20

modules/mgmt-power-remediation-baremetal-about.adoc

@@ -14,7 +14,7 @@ Instead of reprovisioning the nodes, power-based remediation uses a power contro
 Power-based remediation provides the following capabilities:
 
 * Allows the recovery of control plane nodes
-* Reduces the risk data loss in hyperconverged environments
+* Reduces the risk of data loss in hyperconverged environments
 * Reduces the downtime associated with recovering physical machines
 
 [id="machine-health-checks-bare-metal_{context}"]
@@ -23,15 +23,17 @@ Power-based remediation provides the following capabilities:
 Machine deletion on bare metal cluster triggers reprovisioning of a bare metal host.
 Usually bare metal reprovisioning is a lengthy process, during which the cluster
 is missing compute resources and applications might be interrupted.
-To change the default remediation process from machine deletion to host power-cycle,
-annotate the `MachineHealthCheck` resource with the
+
+There are two ways to change the default remediation process from machine deletion to host power-cycle:
+
+. Annotate the `MachineHealthCheck` resource with the
 `machine.openshift.io/remediation-strategy: external-baremetal` annotation.
+. Create a `Metal3RemediationTemplate` resource, and refer to it in the `spec.remediationTemplate` of the `MachineHealthCheck`.
 
-After you set the annotation, unhealthy machines are power-cycled by using
-BMC credentials.
+After using one of these methods, unhealthy machines are power-cycled by using Baseboard Management Controller (BMC) credentials.
 
 [id="mgmt-understanding-remediation-process_{context}"]
-== Understanding the remediation process
+== Understanding the annotation-based remediation process
 
 The remediation process operates as follows:
 
@@ -47,13 +49,30 @@ The remediation process operates as follows:
 If the power operations did not complete, the bare metal machine controller triggers the reprovisioning of the unhealthy node unless this is a control plane node or a node that was provisioned externally.
 ====
 
+[id="mgmt-understanding-metal3-remediation-process_{context}"]
+== Understanding the metal3-based remediation process
+
+The remediation process operates as follows:
+
+. The MachineHealthCheck (MHC) controller detects that a node is unhealthy.
+. The MHC creates a metal3 remediation custom resource for the metal3 remediation controller, which requests to power-off the unhealthy node.
+. After the power is off, the node is deleted, which allows the cluster to reschedule the affected workload on other nodes.
+. The metal3 remediation controller requests to power on the node.
+. After the node is up, the node re-registers itself with the cluster, resulting in the creation of a new node.
+. After the node is recreated, the metal3 remediation controller restores the annotations and labels that existed on the unhealthy node before its deletion.
+
+[NOTE]
+====
+If the power operations did not complete, the metal3 remediation controller triggers the reprovisioning of the unhealthy node unless this is a control plane node or a node that was provisioned externally.
+====
+
 [id="mgmt-creating-mhc-baremetal_{context}"]
 == Creating a MachineHealthCheck resource for bare metal
 
 .Prerequisites
 
 * The {product-title} is installed using installer-provisioned infrastructure (IPI).
-* Access to Baseboard Management Controller (BMC) credentials (or BMC access to each node)
+* Access to BMC credentials (or BMC access to each node).
 * Network access to the BMC interface of the unhealthy node.
 
 .Procedure
@@ -65,7 +84,7 @@ If the power operations did not complete, the bare metal machine controller trig
 $ oc apply -f healthcheck.yaml
 ----
 
-.Sample `MachineHealthCheck` resource for bare metal
+.Sample `MachineHealthCheck` resource for bare metal, annotation-based remediation
 [source,yaml]
 ----
 apiVersion: machine.openshift.io/v1beta1
@@ -105,6 +124,52 @@ spec:
 The `matchLabels` are examples only; you must map your machine groups based on your specific needs.
 ====
 
+.Sample `MachineHealthCheck` resource for bare metal, metal3-based remediation
+[source,yaml]
+----
+apiVersion: machine.openshift.io/v1beta1
+kind: MachineHealthCheck
+metadata:
+  name: example
+  namespace: openshift-machine-api
+spec:
+  selector:
+    matchLabels:
+      machine.openshift.io/cluster-api-machine-role: <role>
+      machine.openshift.io/cluster-api-machine-type: <role>
+      machine.openshift.io/cluster-api-machineset: <cluster_name>-<label>-<zone>
+  selector:
+    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+    kind: Metal3RemediationTemplate
+    name: metal3-remediation-template
+    namespace: openshift-machine-api
+  unhealthyConditions:
+  - type:    "Ready"
+    timeout: "300s"
+----
+
+.Sample `Metal3RemediationTemplate` resource for bare metal, metal3-based remediation
+[source,yaml]
+----
+apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+kind: Metal3RemediationTemplate
+metadata:
+  name: metal3-remediation-template
+  namespace: openshift-machine-api
+spec:
+  template:
+    spec:
+      strategy:
+        type: Reboot
+        retryLimit: 1
+        timeout: 5m0s
+----
+
+[NOTE]
+====
+The `matchLabels` are examples only; you must map your machine groups based on your specific needs. The `annotations` section does not apply to metal3-based remediation. Annotation-based remediation and metal3-based remediation are mutually exclusive.
+====
+
 [mgmt-troubleshooting-issue-power-remediation_{context}]
 == Troubleshooting issues with power-based remediation