Rabbitmq query management

[paywagh11]

Hi All,

I have a Highly available 3 node Rabbitmq cluster in AKS.
There, I have multiple queries which by default are connected to master node and it's copies will be available in mirroring nodes.

But the issue occurs when pods restarts. Whenever the master pod restarts, all the master queues are available/linked on master pod(single pod). Even though one of the mirror will take its place with all the master queues, there will be some fraction sort of delay which causes the queues to be down as well.

How can we rebalance the queues in the pod?
How to avoid this issue?

[mkuratczyk]

Perhaps the answer to your question is rabbitmq-queues rebalance, but it's hard to tell as your question is very vague.

If rabbitmq-queues rebalance is not what you are looking for, please clarify:

• when you said "query", did you mean "queue"?
• there is no such thing as "master pod" or "master node" in RabbitMQ. A specific queue will have a leader (aka master) on a certain node, but the nodes in the cluster are equal and each can host leaders (masters) and followers
• there is no such thing as "master queue"

The most precise and easiest to understand way of asking this kind of questions is to simply show the commands you perform, the output you get and to point out where that output differs from your expectations (screenshots from the Management UI are ok as well).

[paywagh11]

Hello. Thanks for the update.
Sorry but I can't share the outputs due to organisation compliance issue.

Let me rephrase my query more properly once again.
We have a HA 3 node Rabbitmq cluster in AKS. We are trying load testing on this RMQ as per our application needs using jmeter.
We have created multiple queues and they were balanced across the nodes and this test was kept running for several hours. After some 3-4 hours, all the queues came under same node. Here when we checked for pod restart, no pod was restarted in the logs .

What could be the issue here?

Also we ran the Rabbitmq -queue rebalance command as well, but got empty output.

[mkuratczyk]

Leaders only change when requested (rebalanced) or when issue occur (which should create some logs). If you don't think pods were restarted, check the logs. Erlang may restart processes within the runtime, so it's possible that a queue was restarted without the whole pod being restarted (but that means the queue crashed, which should be investigated).

Please also provide the version and queue type if you want further help.

[dormanze]

If you are a temporary queue, you can forcibly close the connection from the server to randomly re-allocate the connection to other nodes. If you're a persistent queue, there's no way.
Run the following command on the node with high load:

rabbitmqctl close_all_connections 'rebalance'

If the most effective way is to delete all queues and let the client re-create them, the initial created queues follow the principle of random allocation. Mirroring queues are essentially active/standby principles, so it is inevitable that the masters are clustered on the same node.

[michaelklishin]

You seems to be reinventing rabbitmq-queues rebalance that quorum queues have supported for a while.

Classic mirrored queues are deprecated and will be removed in early 2024 starting with RabbitMQ 4.0.