Long running Queue list in rabbitmq GUI since migrating to Quorum queues in 3.11.x

[ds2105]

Starting with 3.11.1 we switched most of our queues from Classic to Quorum. The total number of queues and the ha-params configuration are almost the same. Its a three node cluster. We noticed that the time to list the queues in rabbitmq GUI did much increases. It takes round about a minute to get a result of queues lists. The number of queues we use are between 15 K up to 25 K. This effect doesn' t changes with any rabbitmq updates even there are made any performance benefits for Quorum queues. The actual version we are using is 3.11.17.

[michaelklishin]

This sounds like a duplicate of #9518.

Are there any currently down cluster peers or former cluster members that could have been removed?

[ds2105]

No, it doesn't looks like duplicate to #9518. There is no timeout for queues on one of the cluster members. The whole output takes a long time and is much longer than it was for classic queues.

[ds2105]

one example:
rabbitmq-queues quorum_status --vhost PROD WK.CUPPS.AMADEUS.AMM.A001.DLH.ACUSUSER.AMMM0CKA0002.7.1.FL.A.AMM.T.X.CKI.A0002
Status of quorum queue WK.CUPPS.AMADEUS.AMM.A001.DLH.ACUSUSER.AMMM0CKA0002.7.1.FL.A.AMM.T.X.CKI.A0002 on node rabbit@10-244-108-93 ...

rabbitmq-queues check_if_node_is_quorum_critical
Checking if node rabbit@10-244-108-91 is critical for quorum of any queues/streams ...
Node rabbit@10-244-108-91 reported no queues/streams with minimum quorum

rabbitmq-queues check_if_node_is_quorum_critical
Checking if node rabbit@10-244-108-92 is critical for quorum of any queues/streams ...
Node rabbit@10-244-108-92 reported no queues/streams with minimum quorum

rabbitmq-queues check_if_node_is_quorum_critical
Checking if node rabbit@10-244-108-93 is critical for quorum of any queues/streams ...
Node rabbit@10-244-108-93 reported no queues/streams with minimum quorum

[kjnilsson]

With that many queues I could well imagine it getting a bit slow compared to classic queues. What made you use 15k Quorum queues?

The code I suspect is responsible is rabbit_quorum_queue:i/2 which ends up doing a few more calls and rpcs than is strictly necessary and could well be optimised. A profiling capture using perf would help us narrow down and possible optimise the right bit of code faster.

[ds2105]

Thank you Michael, that 's why we will have configured 8 cores initial and with increasing allowed cores until 16 max in k8s environment. That we figured out while mass testing with 40K QQs. The highest resources are used while creating all QQs from scratch. If all QQs are already created the used resources are lower.

[ds2105]

By the way - what does a snapshot index "Undefined" mean. I didn't find a explanation for that. Thank you!

[michaelklishin]

A quorum queue may or may not have any log snapshots at the time when these metrics are requested.

[ds2105]

OK, thank you. So it isn't an error than a valid value. Understood.

[michaelklishin]

We don't have enough information to tell that but very likely it's just the state of the snapshot(s) (there isn't any)