Number of queues bound to x-consistent-hash exchange affects publishing performance a lot

[mwilniewiec]

Describe the bug

Hi, It more like a performance optimization request or they may be some more or less obvious bug underneath:

Recently im playing a lot with x-consistent-hash (RabbitMQ 3.12.4) and I tried to bind 64 queues to it.
I found out that my publishing performance degraded a lot (comparing to 4 queues).
I separated a simple benchmark for this issue:

Im publishing 50 000 persistent messages, 100 in parallel from node.js (amplib) to an x-consistent-hash with some queues bound to it. I can see that messages are evenly distributed but performance is my concern.

These are the results:
1 queue: Time elapsed: 4843 ms (10324 msg/sec)
4 queues: Time elapsed: 3230 ms (15480 msg/sec)
8 queues: Time elapsed: 3988 ms (12538 msg/sec)
16 queues: Time elapsed: 5222 ms (9575 msg/sec)
32 queues: Time elapsed: 10355 ms (4829 msg/sec)
64 queues: Time elapsed: 22483 ms (2224 msg/sec)
128 queues: Time elapsed: 20781 ms (2406 msg/sec)
256 queues: Time elapsed: 44361 ms (1127 msg/sec)

Reproduction steps

1. Create a consistent hash exchange
2. bind 8 queues to it and benchmark publishing performance
3. bind 64 queues to it and compare the publishing performance

Expected behavior

I can imagine that number of queues can somehow affect the publishing performance but a linear degradation seems to be odd.

Additional context

No response

[michaelklishin]

@mwilniewiec if you consider that every queue gets a complete copy of the message, this is not odd at all.

If you want a mass fanout, use streams.

[michaelklishin]

Well, that'd be my primary guess: routing to N queues with this particular exchange should not be all that common in practice. But it depends on the routing key distribution.

In any case, distribution a flow of messages across N queues for parallel consumption is something that streams and in particular superstreams do very well in modern versions. With a couple of orders of magnitude higher throughput and without any decrease in consumer throughput when you need a mass fanout.