Clustering guide: rework the sections on cluster member removal and node reset

Author: michaelklishin

docs/clustering.md

@@ -975,51 +975,52 @@ rabbitmqctl cluster_status
 # => ...done.
 ```
 
-## Forcing Node Boot in Case of Unavailable Peers {#forced-boot}
 
-In some cases the last node to go
-offline cannot be brought back up. It can be removed from the
-cluster using the `forget_cluster_node` [rabbitmqctl](./cli) command.
+## How to Remove a Node from the Cluster {#removing-nodes}
 
-Alternatively `force_boot` [rabbitmqctl](./cli) command can be used
-on a node to make it boot without trying to sync with any
-peers (as if they were last to shut down). This is
-usually only necessary if the last node to shut down or a
-set of nodes will never be brought back online.
+Sometimes it is necessary to remove a node from the cluster.
+
+The sequence of actions will be slightly different for the following
+most common scenarios:
 
-## Breaking Up a Cluster {#removing-nodes}
+ * The node is online and reachable
+ * The node is offline and cannot be recovered
 
-Sometimes it is necessary to remove a node from a
-cluster. The operator has to do this explicitly using a
-`rabbitmqctl` command.
+In addition, if the cluster [peer discovery mechanisms](./cluster-formation)
+support node health checks and [forced removal of nodes](./cluster-formation#node-health-checks-and-cleanup) not known to the discovery backend.
 
-Some [peer discovery mechanisms](./cluster-formation)
-support node health checks and forced
-removal of nodes not known to the discovery backend. That feature is
-opt-in (deactivated by default).
+That feature is opt-in (deactivated by default).
 
-We first remove `rabbit@rabbit3` from the cluster, returning it to
-independent operation. To do that, on `rabbit@rabbit3` we
-stop the RabbitMQ application, reset the node, and restart the
-RabbitMQ application.
+Continuing with the three node cluster example used in this guide,
+let's demonstrate how to remove `rabbit@rabbit3` from the cluster, returning it to
+independent operation.
+
+### Removal of a Reachable Node
+
+First step before removing a node from the cluster is to stop it:
 
 ```bash
 # on rabbit3
 rabbitmqctl stop_app
 # => Stopping node rabbit@rabbit3 ...done.
+```
 
-rabbitmqctl reset
-# => Resetting node rabbit@rabbit3 ...done.
-rabbitmqctl start_app
-# => Starting node rabbit@rabbit3 ...done.
+Then use `rabbitmqctl forget_cluster_node` on another node
+and specify the node to remove as **the first positional argument**:
+
+```bash
+# on rabbit2
+rabbitmqctl forget_cluster_node rabbit@rabbit3
+# => Removing node rabbit@rabbit3 from cluster ...
 ```
 
-Note that it would have been equally valid to list
-`rabbit@rabbit3` as a node.
+Running the
 
+```shell
+rabbitmq-diagnostics cluster_status
+```
 
-Running the <i>cluster_status</i> command on the nodes confirms
-that `rabbit@rabbit3` now is no longer part of
+command on the nodes confirms that `rabbit@rabbit3` now is no longer part of
 the cluster and operates independently:
 
 ```bash
@@ -1036,17 +1037,32 @@ rabbitmqctl cluster_status
 # => [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2]}]},
 # =>  {running_nodes,[rabbit@rabbit1,rabbit@rabbit2]}]
 # => ...done.
+```
+
+Now node `rabbit@rabbit3` can be decomissioned to reset and started as
+a standalone node:
+
 
+```shell
 # on rabbit3
+rabbitmqctl reset
+
+rabbitmqctl start_app
+# => Starting node rabbit@rabbit3 ...
+
 rabbitmqctl cluster_status
 # => Cluster status of node rabbit@rabbit3 ...
 # => [{nodes,[{disc,[rabbit@rabbit3]}]},{running_nodes,[rabbit@rabbit3]}]
 # => ...done.
 ```
 
-We can also remove nodes remotely. This is useful, for example, when
-having to deal with an unresponsive node. We can for example remove
-`rabbit@rabbit1` from `rabbit@rabbit2`.
+Nodes can be removed remotely, that is, from a different host, as long as CLI tools
+on said host can [connect and authenticate](./cli) to the target node.
+
+This can useful, for example, when having to deal with a host that cannot be accessed.
+
+In the rest of this example, `rabbit@rabbit1` will be removed from its remaining
+two node cluster with `rabbit@rabbit2`:
 
 ```bash
 # on rabbit1
@@ -1059,16 +1075,32 @@ rabbitmqctl forget_cluster_node rabbit@rabbit1
 # => ...done.
 ```
 
-Note that `rabbit1` still thinks it's clustered with
+### Removal of Stopped Nodes and Their Revival
+
+:::important
+
+A node that was removed from the cluster when stopped with `rabbitmqctl stop_app`
+must be either reset or decomissioned. If started without a reset,
+it won't be able to rejoin its original cluster.
+
+:::
+
+At this point `rabbit1` still thinks it is clustered with
 `rabbit2`, and trying to start it will result in an
-error. We will need to reset it to be able to start it again.
+error because the rest of the cluster no longer considers it to be a known member:
 
 ```bash
 # on rabbit1
 rabbitmqctl start_app
 # => Starting node rabbit@rabbit1 ...
 # => Error: inconsistent_cluster: Node rabbit@rabbit1 thinks it's clustered with node rabbit@rabbit2, but rabbit@rabbit2 disagrees
+```
+
+In order to completely detach it from the cluster, such
+stopped node must be reset:
+
 
+```shell
 rabbitmqctl reset
 # => Resetting node rabbit@rabbit1 ...done.
 
@@ -1078,7 +1110,7 @@ rabbitmqctl start_app
 ```
 
 The `cluster_status` command now shows all three nodes
-operating as independent RabbitMQ brokers:
+operating as independent RabbitMQ nodes (single node clusters):
 
 ```bash
 # on rabbit1
@@ -1117,18 +1149,48 @@ rabbitmqctl start_app
 # => Starting node rabbit@rabbit2 ...done.
 ```
 
+### Removal of Unresponsive Queues
+
+When target node is not running, it can still be removed from the cluster using
+using `rabbitmqctl forget_cluster_node`:
+
+```bash
+# Tell rabbit@rabbit1 to permanently remove rabbit@rabbit2
+rabbitmqctl forget_cluster_node -n rabbit@rabbit1 rabbit@rabbit2
+# => Removing node rabbit@rabbit1 from cluster ...
+# => ...done.
+```
+
+### What Happens to Quorum Queue and Stream Replicas?
+
+When a node is removed from the cluster using CLI tools, all [quorum queue](./quorum-queues#replica-management)
+and [stream replicas](./streams#replica-management) on the node will be removed,
+even if that means that queues and streams would temporarily have an even (e.g. two) replicas.
+
+### Node Removal is Explicit (Manual) or Opt-in
+
 Besides `rabbitmqctl forget_cluster_node` and the automatic cleanup of unknown nodes
 by some [peer discovery](./cluster-formation) plugins, there are no scenarios
 in which a RabbitMQ node will permanently remove its peer node from a cluster.
 
-### How to Reset a Node {#resetting-nodes}
 
-Sometimes it may be necessary to reset a node (wipe all of its data) and later make it rejoin the cluster.
-Generally speaking, there are two possible scenarios: when the node is running, and when the node cannot start
-or won't respond to CLI tool commands e.g. due to an issue such as [ERL-430](https://bugs.erlang.org/browse/ERL-430).
+
+## How to Reset a Node {#resetting-nodes}
+
+:::danger
 
 Resetting a node will delete all of its data, cluster membership information, configured [runtime parameters](./parameters),
-users, virtual hosts and any other node data. It will also permanently remove the node from its cluster.
+users, virtual hosts and any other node data. It will also alter its internal identity.
+
+:::
+
+Sometimes it may be necessary to reset a node (what specifically this means, see below),
+and later make it rejoin the cluster as a new node.
+
+Generally speaking, there are two possible scenarios: when the node is running, and when the node cannot start
+or won't respond to CLI tool commands for any reason.
+
+### Reset a Running and Responsive Node
 
 To reset a running and responsive node, first stop RabbitMQ on it using `rabbitmqctl stop_app`
 and then reset it using `rabbitmqctl reset`:
@@ -1141,16 +1203,47 @@ rabbitmqctl reset
 # => Resetting node rabbit@rabbit1 ...done.
 ```
 
+:::info
+
+If the reset node is online and its cluster peers are reachable, the node
+will first try to permanently remove itself from its cluster.
+
+:::
+
+### Reset an Unresponsive Node
+
 In case of a non-responsive node, it must be stopped first using any means necessary.
 For nodes that fail to start this is already the case. Then [override](./relocate)
-the node's data directory location or [re]move the existing data store. This will make the node
+the node's data directory location or remove the existing data store. This will make the node
 start as a blank one. It will have to be instructed to [rejoin its original cluster](#cluster-formation), if any.
 
-A node that's been reset and rejoined its original cluster will sync all virtual hosts, users, permissions
-and topology (queues, exchanges, bindings), runtime parameters and policies. [Quorum queue](./quorum-queues)
-contents will be replicated if the node will be selected to host a replica.
+### Resetting a Node to Re-add It as a Brand New Node to Its Original Cluster
+
+A reset node that was [removed from the cluster](#removing-nodes) can be re-added to its original
+cluster as a brand new node.
+
+In that case it will sync all virtual hosts, users, permissions and topology (queues, exchanges, bindings),
+runtime parameters and policies.
+
+For [quorum queue](./quorum-queues) and [stream](./streams) contents to be replicated to the new [re]added node,
+the node must be added to the list of nodes to place replicas on using `rabbitmq-queues grow`.
+
 Non-replicated queue contents on a reset node will be lost.
 
+
+## Forcing Node Boot in Case of Unavailable Peers {#forced-boot}
+
+In some cases the last node to go
+offline cannot be brought back up. It can be removed from the
+cluster using the `forget_cluster_node` [rabbitmqctl](./cli) command.
+
+Alternatively `force_boot` [rabbitmqctl](./cli) command can be used
+on a node to make it boot without trying to sync with any
+peers (as if they were last to shut down). This is
+usually only necessary if the last node to shut down or a
+set of nodes will never be brought back online.
+
+
 ## Upgrading clusters {#upgrading}
 
 You can find instructions for upgrading a cluster in [the upgrade