Does RabbitMQ Sharding Plugin support Quorum Queue

[faithfull]

Hello,

I recently tried the rabbitmq sharding plugin(x-modulus-hash) and found that the shard queues it create are of type classic. Can you please clarify if it support quorum queues ? if yes then how to configure sharding plugin to create quorum queues instead ?

Regards,

[mkuratczyk]

Please read
https://github.com/rabbitmq/rabbitmq-server/tree/main/deps/rabbitmq_sharding#sharding-and-queue-replication

If you still think you need this, please explain what you want to achieve.

[michaelklishin]

Considering what the goal of the sharding plugin is, using quorum queues for shards doesn't make much sense. If you are looking to a similar kind of "logical queue with parallelism", you can achieve that with N queues of any type plus the consistent hashing exchange.

Or justu se superstreams with a RabbitMQ stream protocol client.

[ckfngod]

Why doesn't it make sense? If "N queues of any type plus the consistent hashing exchange" using quorum queues works then why isn't it valid for the sharding plugin to do the same? I thought the sharding plugin was basically just the consistent hashing plugin + automatic shard queue creation / binding. Is there a difference between how queues are sharded in the consistent hashing plugin vs the sharding plugin?

[michaelklishin]

For starters, you are comparing an exchange type to a really non-trivial "logical layer" between a consumer and a "regular" queues.

The sharding plugin does way more than just providing an exchange type similar to the consistent hashing exchange. It overrides how the basic.consume operation in the protocol works by attaching consumers to specific "subqueues". Those subqueues in theory can be anything but in practice, it would only make any kind of sense if you had, say, 7+ nodes with 3 QQ replicas.

Most importantly, the sharding plugins is well past its prime in terms of relevancy. Quorum queues, classic queues, and in particular streams all offer better throughput than around 2016 when that plugin was designed.

The consistent hashing exchange or the x-module exchange alone can help maintain "a logical queue" without any complexity of the sharding plugin.

Streams are orders of magnitude more efficient and allow for repeatable reads, and have plenty of parallelism built into the protocol, as long as you have enough nodes and consumers online.

The sharding plugin and quorum queues would be a fairly odd combination that technically could work but QQs will likely eliminate most of the benefits of sharding simply because they are always replicated.

If you want to use QQs and this kind of distribution using the hashing exchanges, you can do so without the sharding plugin. Or you can simply use streams where parallelism and efficiency are most important.

[ckfngod]

Gotcha so this was really just a way to provide mirroring-like behaviour by manually distributing a queue across nodes? Why does it make more sense for "7+ nodes with 3 QQ replicas"? Is it because there isn't already a replica on every node and that would allow some room for this kind of partitioning?

The consistent hash exchange with quorum queues fits my needs but the only issue is having to deal with consumer balancing across instances of my application so I was really hoping the sharding plugin would be able to handle that for me. Is there another plugin or some other way I can get the broker to handle consumer balancing?

I've also taken a look at super streams but it doesn't look like there is a mature JS client for it yet.

[michaelklishin]

Not mirroring-like behavior but a partitioning mechanism similar to what RabbitMQ stream protocol has.

Replication can be added on top of that. However, replication also reduces the positive effect of parallelisation. Streams give you an optimal combination of both.