When using peer discovery, nodes must be restarted to forget a cluster node

[lukebakken]

Reported here:
https://groups.google.com/g/rabbitmq-users/c/yr--HAlasOE

This project demonstrates setting up a cluster using classic peer discovery:
https://github.com/lukebakken/rabbitmq-users-yr--HAlasOE

This script shows the steps necessary to forget a cluster node (rmq0 in this case):
https://github.com/lukebakken/rabbitmq-users-yr--HAlasOE/blob/main/remove-rmq0.sh

Note that nodes must be restarted (after commenting out the relevant config lines) in order for the peer discovery configuration to be cleared from RabbitMQ. forget_cluster_node and stop_app/start_app is not sufficient.

cc @michaelklishin - is this expected?

[michaelklishin]

Peer discovery is only performed by freshly booting nodes early in the boot process. It does not change cluster membership.

Node removal generally assumes that the node is stopped or won't be coming back.

A restarted node, if it reads the same rabbitmq.conf, would rejoin its cluster early on boot. This may be what confuses you. There should be debug log messages around peer discovery activity.

In other words, a node without any peer discovery information won't know how to rejoin the cluster but a node that is set up to perform peer discovery will, if it is restarted for any reason at any moment after I was removed from the original cluster.

[lukebakken]

A restarted node, if it reads the same rabbitmq.conf, would rejoin its cluster early on boot

I'm making sure to comment out the appropriate lines in rabbitmq.conf before running stop_app / start_app or reset. I was surprised to see that none of these operations (even reset) re-reads configuration. Should it? The only way that I found was to completely stop the node and start it again.