Unable to add/remove members from quorum queue with no active members

[pardahlman]

We have a problem with some of our quorum queues that we are not able to solve. I've asked the question in the rabbitmq user group without any response. If you can find the time to point me in the right direction it would be greatly appreciated

We have a cluster with three nodes running RabbitMQ 3.10.19 on ERL 25.3. There are a number of quorum queues where all members are offline (they have been decommissioned). We want these queues to only have members from the currently active nodes, and we are OK with recreating the queues. So far, we've tried the following:

Remove the queues using the management UI. This operation fails and the UI becomes unresponsive

Remove member from the queue using the rabbitmq-queues CLI. This operation fails with the following error message >"no_more_servers_to_try" and 3x timeouts

Adding a new (existing) member to the queue also fails, with "no_more_servers_to_try" and 3x nodedown

Attached are screenshots from the management UI.

Any thought on how we can interact with these queues?

Best regards,
Pär

[lukebakken]

I will follow up on the mailing list. Please note that it has only been 3 days since you asked. The RabbitMQ team is small and we endeavor to reply to questions within a week.

[pardahlman]

Thanks, Luke! Apologies if I created an issue prematurely!

[lukebakken]

PS I thought your github handle was familiar. Thank you for the docker cluster project you shared. I have used it as a starting point for my own use of Docker and RabbitMQ - https://github.com/lukebakken/docker-rabbitmq-cluster

[pardahlman]

How nice to see that someone else found it useful! 🤩

[michaelklishin]

If a quorum queue does not have "active members" I assume it does not have an elected leader either, which means you cannot make any membership changes.

[pardahlman]

Thanks for the reply! Yes, based on my interaction with the queues via the management UI and rabbitmq-queues CLI no changes seems to be able to be made. However, the queues in question (or rather, queues with those specific queue names) are important so we are looking for ways to either somehow make membership changes or remove and recreate them altogether.

[pardahlman]

The issue was resolved in the rabbitmq-users group by running:

rabbitmqctl eval 'Q = {resource, <<"virtual host name">>, queue, <<"queue name">>}, rabbit_amqqueue:internal_delete(Q, <<"CLI">>).'

As suggested by Luke. Thanks to the entire RabbitMQ team for taking the time to look into this 🙏

[pecigonzalo]

Piggybacking on this reply, I have the same issue on version RabbitMQ 3.10.10. We have 2 nodes that no longer exist, and some queues that have only one member. We can't seem to delete those nodes or add new ones. The only available member shows as pre_vote and while this eval commands returns ok, the members are not removed and no new members can be added.

[pecigonzalo]

I see my mistake here, I understood <<"virtual host name">> to be the node I wanted to remove, since I was hoping no to fully recreate the queue. I understand now that this is actually the vhost.

Is there a way to just remove the 2 broken nodes and keep the healthy one instead of plain deleting the queue?

[michaelklishin]

You can remove quorum queue replica count after you add however many replacement nodes you need, then see rabbitmqctl help forger_cluster_node about how to decommission a node.

[pecigonzalo]

Hey @michaelklishin, thanks for your reply. So you are saying that if 2 out 3 nodes crash or are lost, and Im left with a single node in the quorum, I should be able to add nodes, and then remove nodes?

The problem I found is that doing grow or shrink does not seem to work as it complains that it cant remove/add nodes, since the quorum only has a single node and it stays in "pre-vote" state.

[michaelklishin]

If you permanently lose a majority of nodes in a Raft-based system, your only hope that the tool allows for some kind of forced boot or emergency recovery without a quorum. RabbitMQ does not, although we would not be against introducing this feature eventually. But it is significantly more risky than it sounds, so a couple of attempts were put on pause.

[pecigonzalo]

Sorry for taking a while to get back here and thanks again for your reply. I think it would be great to have such a feature, with the known risks of data loss, as in a scenario like ours:

• We could tolerate the data loss
• We could not, without a significant amount of effort, re-create the nodes with the same identities, etc. so we could restore the queues

This meant our option went from partial data loss to total data loss, as we had to fully re-create the queues.

PS: I know the question here was answered both for the original and my additional one, but I thought its still good to combine this information/conversation in this github discussion.