Quorum queues while ra's not running.

[markusgust]

hello all!

We had some quorum queues crash during a reboot recently, which wasn't a massive deal on its own, however while troubleshooting the instance we noticed that rabbitmqctl eval 'ra:overview().' returned system_not_started. When running rabbitmqctl eval 'ra:start().' all the rest of our quorum queues went down, and wasn't resurrected until we restarted the node. After the reboot we still received system_not_startedwhen evaluatingra:overview()`.

Is this by design? I guess I'm curious how starting raft would crash our current queues. On 3.12.6, one node cluster. (however tested and had the same behaviour on a 3-node cluster).

steps to reproduce for the curious:

1. Create a 1-node cluster on 3.12.6.
2. create a quorum queue
3. using rabbitmq-diagnostics observer -> Plugin -> QQ's (or look at the management console) and confirm the queue is running.
4. evaluate rabbitmqctl eval 'ra:overview(). followed by ra:start() and watch the queue go down. Observer will then show the queue, but no PID.

Thank you.

[michaelklishin]

You assume that quorum queues use the default Ra system which hasn't been the case since 3.10 or so.

But that's what ra:overview/0 and ra:start/0 use. If you want to know the state of a specific Ra system, such as QQs or streams, you must use ra:overview/ and their respective names (by searching for ?RA_SYSTEM across the codebase).

I cannot suggest much without any logs but one thing that is still shared by all Ra systems is a number of metrics, so restarting the default system must have wiped some of them.

[markusgust]

cool - yes that's what I assumed and what you're saying make sense to me. Thank you!

[michaelklishin]

I'm pretty sure it goes without saying but RabbitMQ codebase does not generally assume that someone will be messing with Ra systems using rabbitmqctl eval or a remote shell.