API endpoint for forget_cluster_node

[zaeemarshad]

What

To remove a dead cluster node from rabbitmq, forget_cluster_node has to be called using rabbitmqctl. There's no equivalent API endpoint that offers this functionality.

Use case

Consider a 3-node cluster with nodes A, B, and C with auto-cleanup disabled. Let's say node C is replaced with node B. RabbitMQ will record nodes A,B,C and D as cluster members. The cluster's partition behaviour is pause_minority. If any other node is taken offline temporarily, the cluster will lose majority all nodes will be paused until the offline node comes back online again. In environments where nodes are ephemeral, removing dead nodes from the cluster in a quick time frame is important.

To prevent this from happening, we want to make sure that a node which no longer exists should be removed from the cluster configuration in a timely manner. In our case, when a node is no longer alive i.e. VM is deleted, we trigger an internal event to PubSub for the deletion. This event is picked up by a worker which does a bit of logic check before calling rabbitmqctl forget_cluster_node to remove the dead node from the cluster.

Currently, this requires the worker to run on a RabbitMQ node or has rabbitmqctl installed wherever it runs. If the API endpoint is available, then it can be exposed to something as simple as a cloud function which can call the endpoint and remove the dead node from the cluster without using rabbitmqctl.