Rebalance quorum

[MonicaMagoniCom]

I have a cluster of 4 rabbitmq instances with both quorum and classic queues. When I do:
rabbitmq-queue rebalance all

I get the following message:
[warning] Node :"rabbit@rabbitmq-quorum-server-0.rabbitmq-quorum-nodes.rabbitmq-system-quorum" contains 128 queues, but all have already migrated. Do nothing

What does this mean?
I actually have one instance which only has few queues and with the rebalancing nothing changes.

[kjnilsson]

Quorum queues only place themselves over 3 nodes by default so perhaps there are fewer members on one of your nodes. Why 4 nodes?

…

On Thu, 4 May 2023 at 10:13, MonicaMagoniCom ***@***.***> wrote: I have a cluster of 4 rabbitmq instances with both quorum and classic queues. When I do: rabbitmq-queue rebalance all I get the following message: [warning] Node :" ***@***.***" contains 128 queues, but all have already migrated. Do nothing What does this mean? I actually have one instance which only has few queues and with the rebalancing nothing changes. — Reply to this email directly, view it on GitHub <#8099>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAHFA73TZYR6GM4XTUE4DXENXMJANCNFSM6AAAAAAXVQA4CU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- *Karl Nilsson*

[MonicaMagoniCom]

But if I want to spread the replicas over 4 nodes how can I do it? I mean: replication factor is 3, but nodes are more than 3

[kjnilsson]

A quorum queue declare will use the node used for the declaration + 2 random nodes to spread the quorum queue over. You can manually modify which members go where using the rabbitmq-queues tool. https://www.rabbitmq.com/quorum-queues.html#replica-management

…

On Thu, 4 May 2023 at 12:56, MonicaMagoniCom ***@***.***> wrote: But if I want to spread the replicas over 4 nodes how can I do it? I mean: replication factor is 3, but nodes are more than 3 — Reply to this email directly, view it on GitHub <#8099 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAHFDRSKDJAF7HODGT5ULXEOKNLANCNFSM6AAAAAAXVQA4CU> . You are receiving this because you commented.Message ID: ***@***.*** com>

-- *Karl Nilsson*

[MonicaMagoniCom]

I can try to explain why I'm asking you this: suppose I have a cluster of 3 nodes with quorum queues; suppose that after some time, the traffic and the number of queues increases and the cluster resources usage becomes really high. To prevent customer dissatisfaction and system malfunction, I decide to increase cluster members to 4. How can I use the fourth replica to reduce other replica's load?

Thank you

[kjnilsson]

You can use the grow and shrink commands to move some members of some queues to the 4th node so say half of the queues span, A, B, C and half span B, C, D. However because all nodes need members on B and C you may not see that much benefit from it. If you increase your node count to 6 you can split it better.

…

On Thu, 4 May 2023 at 14:33, MonicaMagoniCom ***@***.***> wrote: I can try to explain why I'm asking you this: suppose I have a cluster of 3 nodes with quorum queues; suppose that after some time, the traffic and the number of queues increases and the cluster resources usage becomes really high. To prevent customer dissatisfaction and system malfunction, I decide to increase cluster members to 4. How can I use the fourth replica to reduce other replica's load? Thank you — Reply to this email directly, view it on GitHub <#8099 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAHFER7DMFLKXCHLP2A23XEOV3FANCNFSM6AAAAAAXVQA4CU> . You are receiving this because you commented.Message ID: ***@***.*** com>

-- *Karl Nilsson*

[kjnilsson]

"all queues need members on B and C" I meant to write.

…

On Thu, 4 May 2023 at 14:41, Karl Nilsson ***@***.***> wrote: You can use the grow and shrink commands to move some members of some queues to the 4th node so say half of the queues span, A, B, C and half span B, C, D. However because all nodes need members on B and C you may not see that much benefit from it. If you increase your node count to 6 you can split it better. On Thu, 4 May 2023 at 14:33, MonicaMagoniCom ***@***.***> wrote: > I can try to explain why I'm asking you this: suppose I have a cluster of > 3 nodes with quorum queues; suppose that after some time, the traffic and > the number of queues increases and the cluster resources usage becomes > really high. To prevent customer dissatisfaction and system malfunction, I > decide to increase cluster members to 4. How can I use the fourth replica > to reduce other replica's load? > > Thank you > > — > Reply to this email directly, view it on GitHub > <#8099 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAJAHFER7DMFLKXCHLP2A23XEOV3FANCNFSM6AAAAAAXVQA4CU> > . > You are receiving this because you commented.Message ID: > ***@***.*** > com> > -- *Karl Nilsson*

-- *Karl Nilsson*

[MonicaMagoniCom]

How can I move replica from one node to another one?
As far as I understood, grow will add a new replica on the give node, while shrink will remove ALL the replica on tne node

[mkuratczyk]

Rebalancing only changes the leadership, not membership. Most likely you had a 3 node cluster and then added the 4th. None of the queues has a member on the 4th node and rebalancing won't change that.

You might be interested in:
https://www.rabbitmq.com/rabbitmq-queues.8.html -> grown and add_member commands

There are also some things coming in 3.13 that could be helpful in such cases:
https://www.youtube.com/watch?v=_klVJpiimIU

[MonicaMagoniCom]

Yes, that is my scenario. With grow and add_mumber the replica factor would become 4.

Also, I created a new queue and it took as leader the 4th node. However if it try to rebalance now, the 4th node continues to be a leader only of 1queue.

[mkuratczyk]

You can grow the queue(s) and then shrink them / remove old members.

[MonicaMagoniCom]

Ok, I thought there was a more automatic way.

[mkuratczyk]

The 3.13 feature I share a video about could be the basis for this in the future.