draft

Author: kaitlynmichael

content/operate/kubernetes/active-active/recover.md

@@ -1,5 +1,5 @@
 ---
-Title: Remove failed participating cluster
+Title: Remove an unresponsive participating cluster
 alwaysopen: false
 categories:
 - docs
@@ -9,43 +9,120 @@ description:
 linkTitle: Remove RERC
 weight: 98
 ---
+  
+When a RedisEnterpriseRemoteCluster (RERC) isn't communicating with other participating clusters, the RedisEnterpriseActiveActiveDatabase(REAADB) will enter an error state. To remedy this, you can remove the unresponsive RERC from the Active-Active database configuration.
 
 ## Prerequisites
 
-## Identify failed RERC
+We strongly recommend backing up your cluster before proceeding with the following steps.
 
-1. 
+{{<warning>}}Proceed with caution. Ensure you are running the following command from the correct instance. Verify the status of the RERCs and REAADB before and after each step.{{</warning>}}
 
-```sh
-kubectl get rerc
-```
+The followings steps use these example values. Remember to substitute these with your own values.
 
-2. 
+- RERCs: `new-york-1` (cluster name `rec1`), `new-york-2` (`rec2`) and `boston-1` (`rec3`)
+- `<REAADB_NAME>`: `example-aadb-1`
+- `<INSTANCE_ID>`: `3`
+-`<REAADB_GUID>`: `968d586c-e12d-4b8f-8473-42eb88d0a3a2`
 
-```sh
-kubectl get reaadb <REAADB_NAME> -o yaml
-```
+## Identify unresponsive RERC
 
-## Remove failed RERC
+1. From a functioning cluster, check the status of all your RERCs.
 
-3. 
+  ```sh
+  kubectl get rerc
+  ```
 
-```sh
-crdb-cli crdb remove-instance --crdb-guid <REAADB_GUID> --instance-id <INSTANCE_ID> --force
-```
+  The unresponsive RERC will show an `Error` status (`new-york-2` in this example).
+
+  ```sh
+  NAME          STATUS   SPEC STATUS   LOCAL
+  new-york-1     Active   Valid         true
+  new-york-2     Error    Valid         false
+  boston-1       Active   Valid         false
+  ```
+
+2. Find the `guid` of the REAADB and the instance `id` of the unresponsive cluster in the REAADB YAML file.
+
+  ```sh
+  kubectl get reaadb <REAADB_NAME> -o yaml
+  ```
+  
+  <pre><code>
+  apiVersion: app.redislabs.com/v1alpha1
+  kind: RedisEnterpriseActiveActiveDatabase
+  metadata:
+  ....
+      name: example-aadb-1
+      namespace: ns1
+  ....
+  spec:
+      ....
+      participatingClusters:
+      - name: new-york-1
+      - name: new-york-2
+      - name: boston-1
+      redisEnterpriseCluster:
+        name: rec1
+    status:
+      <b>guid: 968d586c-e12d-4b8f-8473-42eb88d0a3a2</b>
+      participatingClusters:
+     - id: 1
+        name: new-york-1
+      - <b>id: 3</b>
+          name: new-york-2
+          replicationStatus: up
+      - id: 2
+          name: boston-1
+          replicationStatus: up
+      ....
+      status: error
+    </code></pre>
+
+## Remove RERC
+
+3. From a working cluster, remove the unresponsive RERC from the Active-Active database.
 
-4.
+  {{<warning>}}Verify you are removing the correct instance before running this command.{{</warning>}}
 
-5. 
+  ```sh
+  crdb-cli crdb remove-instance --crdb-guid <REAADB_GUID> --instance-id <INSTANCE_ID> --force
+  ```
+
+4. Wait a few minutes until the REAADB status remains `active` consistently.
+
+  ```sh
+  kubectl get reaadb example-aadb-1
+  NAME              STATUS   SPEC STATUS   LINKED REDBS   REPLICATION STATUS
+  example-aadb-1    active   Valid                        up
+  ```
+
+5. Edit the REAADB to remove the unresponsive RERC from the `spec.particpatingClusters` section.
+
+  ```sh
+  kubectl edit reaadb <REAADB_NAME>
+  ```
+
+6. Wait a few minutes until the REAADB status remains `active` consistently.
+
+  ```sh
+  kubectl get reaadb example-aadb-1
+  NAME              STATUS   SPEC STATUS   LINKED REDBS   REPLICATION STATUS
+  example-aadb-1    active   Valid  
+  ```
+
+7. From each working participating cluster, repeat steps 5 and 6 to remove the RERC from the REAADB.
+
+## Optional: Purge local data
+
+If you want to delete REAADB data from the unresponsive cluster, you can run the `crdb-cli purge-instance` command from the unresponsive cluster.
+
+{{<warning>}}Use with caution and verify you are purging the correct instance.{{</warning>}}
 
 ```sh
-kubectl edit reaadb <REAADB_NAME>
+crdb-cli crdb purge-instance --crdb-guid  <REAADB_GUID> --instance-id <INSTANCE_ID>
 ```
 
-6. 
+## Next steps
 
-```sh
-kubectl get reaadb example-aadb-1
-NAME              STATUS   SPEC STATUS   LINKED REDBS   REPLICATION STATUS
-example-aadb-1    active   Valid                        up
-```
+After the communication issue on the cluster is resolved, you can add it back to the REAADB configuration by following the steps for [Add a participating cluster]({{<relref "operate/kubernetes/active-active/edit-clusters#add-a-participating-cluster">}}.