Too Many Quorum Queues

[hamzahqt]

Tried Quorum queues on a cluster of 3 nodes
( running RabbitMQ 3.8.28 and erlang 24.2 )
the application ( Openstack ) create more than 3000 quorum queues in multiple VHOSTs

after creating the quorum queues on the cluster

• rabbitmqctl cluster_status show a healthy cluster
• rabbitmq-diagnostics check_virtual_hosts all running
• rabbitmq-diagnostics discover_peers --timeout 60 works fine
• rabbitmq-diagnostics is_running fully running
• Memory usage and number of open connection within the linux limits ( it seems not related to linux system limits on files, .. )

But the rabbitmqctl list_unresponsive_queues --vhost <vhost>

• Allot of quorum queues are unresponsive
• Connections to RabbitMQ are refused sometimes
• The system using the MQ cluster is not working

Removing the quroum queues and returning to the Classic Queues ( no replication )

• All queues are responsive
• Connections never get refused

Tried to increase the following configurations

tcp_listen_options.backlog = 8192
tcp_listen_options.nodelay = true
#tcp_listen_options.packet = raw
#tcp_listen_options.reuseaddr = true
tcp_listen_options.exit_on_close = false
tcp_listen_options.keepalive = true
tcp_listen_options.sndbuf = 32768
tcp_listen_options.recbuf = 32768
tcp_listen_options.linger.on = true
tcp_listen_options.linger.timeout = 0
vm_memory_high_watermark.relative = 0.8
#channel_operation_timeout = 60000
raft.wal_max_size_bytes = 1610612736

Am I missing something, or some configuration i missed

[michaelklishin]

Your node may be hovering around the open file handle limit, so no new connections would be possible. Quorum queues use the disk much more frequently than classic queues.

None the of the rabbitmq.conf settings listed above would help with running out of file descriptors.
I'm not sure why you are bumping ra.wal_max_size_bytes.

It may also be out of CPU resources since every queue consumes some, even if it's just to emit metrics or communicate with other replicas.

[hamzahqt]

Thanks @michaelklishin for the quick response

i checked most of the specs related to open file handle limit, and its fine as shown below

ulimit -n 32768
and lsof -p <rabbitmq-process-id> | wc -l shows max 1600 -> 1800 sometimes 2000+
while the problem was still happening

CPU usage uptime load was 1->2 max (The MQ nodes in cluster specs -> CPU(s):24, RAM : 46 each)

rabbitmq-queues quorum_status --vhost <vhost> <quorum-queues>
show all the three nodes in the cluster as followers ( no leader) for some quorum queues

[michaelklishin]

Alright. Give 3.9.14 or 3.10.0-rc.1 a try. 3.8 is out of general support and a lot of things have changed around quorum queues since 2019 when they first shipped.

They would still use more file handles and more resources than non-mirrored classic queues
but especially in 3.10, some of their footprint has been reduced.

And there is nothing wrong with using quorum queues for some things, a non-mirrored classic queues when it's enough (replication is not needed).

[artificial-intelligence]

assuming you already use a systemd based deployment or even a docker based deployment you might want to check systemds and dockers ulimit values as well, as these are managed differently.