Quorum queue does not elect a new leader in my test

[skry-dev]

Hi,

I have 5 node RabbitMQ cluster. I use 5 nodes quorum queues. When I shut down 2 of nodes some of my queues does not work.
I expect one of the 3 remaining nodes to become the leader and the queue to remain active. But some queues was disabled.
I have 3 active nodes in quorum but none of them took the queue after shutting down 2 nodes.

When I checked the logs, i see that for disabled queues:

2024-07-19 11:58:11.978788+03:00 [notice] <0.1546.25> queue 'xxx' in vhost '/xx': candidate -> leader in term: 4 machine version: 3
2024-07-19 11:58:11.995210+03:00 [info] <0.621.0> node 'xxx' down: connection_closed
2024-07-19 11:58:12.003249+03:00 [notice] <0.1533.25> queue 'xxx' in vhost '/xx': candidate -> leader in term: 4 machine version: 3
2024-07-19 11:58:12.003302+03:00 [notice] <0.1528.25> queue 'xxx' in vhost '/xx': candidate -> leader in term: 171 machine version: 3
2024-07-19 11:58:12.007838+03:00 [notice] <0.1538.25> queue 'xxx' in vhost '/xx': candidate -> leader in term: 4 machine version: 3
2024-07-19 11:58:12.055837+03:00 [notice] <0.1544.25> queue 'xxx' in vhost '/xx': candidate -> leader in term: 6 machine version: 3
2024-07-19 11:58:13.016159+03:00 [info] <0.621.0> rabbit on node 'xx' down
2024-07-19 11:58:13.030269+03:00 [info] <0.1536.25> queue 'xxx' in vhost '/xx': **Leader monitor down with shutdown, setting election timeout**
2024-07-19 11:58:13.039005+03:00 [info] <0.1540.25> queue 'xxx' in vhost '/xx': **Leader monitor down with shutdown, setting election timeout**
2024-07-19 11:58:17.432142+03:00 [info] <0.621.0> node 'xxx' down: connection_closed

How can I fix it?

[skry-dev]

Hi,

The problem continues,

When I checked logs again, I see calls for election, but no leader selection. In that log for QDAILY voting has been done and granting for rabbit1 node, but the node can not be leader, voting keeps repeating.;

2024-08-01 09:33:54.551568+03:00 [debug] <0.21678.13> queue 'QDAILY' in vhost '/vh1': pre_vote election called for in term 7
2024-08-01 09:33:54.555721+03:00 [debug] <0.21678.13> queue 'QDAILY' in vhost '/vh1': pre_vote granted #Ref<0.4076831643.1152385025.241348> for term 7 votes 1
2024-08-01 09:33:54.556228+03:00 [debug] <0.21678.13> queue 'QDAILY' in vhost '/vh1': pre_vote granted #Ref<0.4076831643.1152385025.241348> for term 7 votes 2
2024-08-01 09:33:54.556481+03:00 [debug] <0.21678.13> queue 'QDAILY' in vhost '/vh1': pre_vote granted #Ref<0.4076831643.1152385025.241348> for term 7 votes 3
2024-08-01 09:33:54.715263+03:00 [debug] <0.21678.13> queue 'QDAILY' in vhost '/vh1': granting pre-vote for {'/vh1_QDAILY','rabbit@rabbit1'} machine version (their:ours:effective) 3:3:3 with last indexterm {274,7} for term 7 previous term 7

[michaelklishin]

We do not have enough information to suggest anything specific. You haven't shared the RabbitMQ version used, logs from all nodes and a set of steps to perform to reproduce this behavior.

The log snippet provided does not really tell anything beyond the fact that a node was down and election timeout (a Raft term) was set.

A quorum queue will not be available if a majority of its replicas is not online. A QQ with three replicas (which is the default regardless of cluster size) two of which were stopped will only have an online minority. Note that it does not matter how many cluster nodes you have, I am talking about QQ replicas.

Having five nodes along is not enough, they all must have replicas placed on them. When nodes are replaced, those replicas must be explicitly removed and the new ones added. This is documented in more detail in the Upgrades guide. This feature in QQs in 3.13 should also be relevant for cases where nodes are added and removed (not stopped without explicit removal).