Merge pull request #50076 from xenolinux/disaster-recovery

BZ2096868: Add steps to restore to a previous cluster state

Author: lpettyjo

modules/dr-restoring-cluster-state.adoc

@@ -284,6 +284,57 @@ etcd-ip-10-0-143-125.ec2.internal                1/1     Running     1
 If the status is `Pending`, or the output lists more than one running etcd pod, wait a few minutes and check again.
 
 .. Repeat this step for each lost control plane host that is not the recovery host.
++
+[NOTE]
+====
+Perform the following step only if you are using `OVNKubernetes` Container Network Interface (CNI) plug-in.
+====
++
+. Restart the Open Virtual Network (OVN) Kubernetes pods on all the hosts.
+
+.. Remove the northbound database (nbdb) and southbound database (sbdb). Access the recovery host and the remaining control plane nodes by using Secure Shell (SSH) and run the following command:
++
+[source,terminal]
+----
+$ sudo rm -f /var/lib/ovn/etc/*.db
+----
+
+.. Delete all OVN-Kubernetes control plane pods by running the following command:
++
+[source,terminal]
+----
+$ oc delete pods -l app=ovnkube-master -n openshift-ovn-kubernetes
+----
+
+.. Ensure that all the OVN-Kubernetes control plane pods are deployed again and are in a `Running` state by running the following command:
++
+[source,terminal]
+----
+$ oc get pods -l app=ovnkube-master -n openshift-ovn-kubernetes
+----
++
+.Example output
+[source,terminal]
+----
+NAME                   READY   STATUS    RESTARTS   AGE
+ovnkube-master-nb24h   4/4     Running   0          48s
+ovnkube-master-rm8kw   4/4     Running   0          47s
+ovnkube-master-zbqnh   4/4     Running   0          56s
+----
+
+.. Delete all `ovnkube-node` pods by running the following command:
++
+[source,terminal]
+----
+$ oc get pods -n openshift-ovn-kubernetes -o name | grep ovnkube-node | while read p ; do oc delete $p -n openshift-ovn-kubernetes ; done
+----
+
+.. Ensure that all the `ovnkube-node` pods are deployed again and are in a `Running` state by running the following command:
++
+[source,terminal]
+----
+$ oc get  pods -n openshift-ovn-kubernetes | grep ovnkube-node
+----
 
 . Delete and recreate other non-recovery, control plane machines, one by one. After these machines are recreated, a new revision is forced and etcd scales up automatically.
 +