RabbitMQ Single Node Latency / Memory / Throughput limits

[JanEggers]

we are currently using RabbitMq single node 3.11 deployment to communicate with our PLCS. The PLCS use MQTT and we use a controller type app that uses amqp that subscribes to all events from the plcs and sends commands to controll them.

We observed some things where we are not sure whats the reason / if there are ways we can improve.

Throughput:
We use MQTT explorer to monitor the system. It subscribes to all messages. 1 running Instance of it fine 1200 msg/sec. But with 2 running instances the outbound traffic reduces to 600 msg/ sec and we see items buildup in the queues. This happens at network utilization of up to 30 mb / sec. But we run GB so there should be room for 5x more traffic without queue buildup. (CPU is at 20%)

are there any limits to rabbit that limit the traffic rate besides network constraints?

Memory usage:

the brokers memory usage increases up to several GB although all Queues are empty. the memory is accumulated in binary references whatever that means (diagnosed with admin site). and after a reboot of the broker we are back to 200 MB memory usage.

Latency:
we added logging to our app that shows roundtrip latency. the pattern is always PLCS sends event. controller sends command. PLC loggs latency bewteen event and command. everything is running in the same local network but the roundtrip times are in a wild range from 40ms to 2 sec.

is this normal? is there any way to improve latency?

[lukebakken]

Please try a 3.12 release candidate, there are huge improvements to RabbitMQ's MQTT support -

https://blog.rabbitmq.com/posts/2023/05/rabbitmq-3.12-performance-improvements/#new-mqtt-implementation-significant-memory-savings-per-connection-supports-millions-of-connections-per-node

cc @ansd

[JanEggers]

just did that. we had to roll back as connections have been shaky: https://github.com/rabbitmq/rabbitmq-server/issues/8251

[ansd]

When I send with much higher rates I see temporarily that ~1k messages queue up in the MQTT Explorer classic queue and that consumer utilisation is around 50%. However, eventually the queue is being emptied. So, it all depends on how fast MQTT explorer can consume the messages.

Long story short is: Right now I don't see any performance issues with RabbitMQ itself.
Make sure your MQTT clients can consume fast enough - if they don't: TCP backpressure is applied to the server preventing the server from sending fast enough resulting in messages being accumulated in the classic queue leading to high end-to-end latency.

[JanEggers]

thanks a lot for your support / advice.

we will try classic_queue.default_version = 2.

rabbit_mqtt_qos0_queue is not viable for us as we should not drop messages

just to clarify:

• if messages are accumulated in one queue because the consumer is not fast enough. does that increase the latency for other queues / cllients.

• we thought about switching to a multi node / cluster configuration to distribute load / reduce latency.
  is this benificial?

[ansd]

rabbit_mqtt_qos0_queue is not viable for us as we should not drop messages

If you must not drop messages, you should use QoS 1.
If you should not drop messages, you could also increase mqtt.mailbox_soft_limit to some higher value (than the current default of 200).

if messages are accumulated in one queue because the consumer is not fast enough. does that increase the latency for other queues / cllients.

No.

we thought about switching to a multi node / cluster configuration to distribute load / reduce latency.
is this benificial?

In general, it could. But I don't think this will solve the issue you are experiencing here: That the MQTT explorer client cannot consume the messages fast enough.

[michaelklishin]

There is a claim made above about nodes "accumulating binaries", with a screenshot that clearly demonstrates that what consumes seemingly well over 90% of memory is classic queues.

They can contain unconsumed messages or not, but they are not binary heap fragments.

Most likely what happens with this workload is that consumers come and go, and leave queues behind. RabbitMQ will not delete them unless those queues are exclusive (not necessarily a good idea) or have TTL configured for them (using a policy, MQTTv3 clients do not have a way of doing that).

So setting a queue TTL of 3-12 hours should help a lot. Note that queues that have consumers will not be deleted due to TTL.

[michaelklishin]

The MQTT client in 3.11, and likely comparably in 3.12, can do 5-10K messages a second per connection. There is a realistic limit to how far that can go with more connections but double digit thousands sounds perfectly achievable with efficient clients.

Network bandwidth is not the only limiting factors. Disk I/O and CPU cores should not be ignored. RabbitMQ will happily use more cores if there are more connections/queues (subscribers in case of MQTT).

You can always try PerfTest and see what your setup is capable of. Yes, it does not use MQTT but it will establish a baseline. PerfTest is a mature, very powerful tool that can simulate a broad range of workloads.

We do not have an equivalent for MQTT but there should be a similar tool in the MQTT community. Use that instead of guessing, and collect metrics in the process.

[pfeifi01]

We have the old format of the config file and we were trying to adjust it to set the classic queue version to 2. We are not perfectly sure if we have adjusted it correctly. Is there any way to check via the RabbitMQ Web Dashboard if the queues have now version 2?

[ansd]

There is in 3.12.0-rc.3 thanks to @gomoripeti : #8241
In the Management UI, click on the classic queue details, there should be a field called queue storage version.

[michaelklishin]

We should expose this via CLI tools, too. I will file an issue.

[JanEggers]

can you please have a look at this repro:
https://github.com/JanEggers/RabbitMqLatencyBenchmark

the more producers there are the worse the latency. i reduced the numbers of messages send the more clients i used so the message rate is constant.

also it seems to be a general issue regardless of the protocol when I use amqp as producer i can also see degrading latency the more producers there are

[JanEggers]

i finally was able to configure queue version 2 but that just brought minor improvements.

[JanEggers]

i updated the benchmark again. I added a dotnet mqtt broker to funnel all messages through a single amqp connection. that way i was able to bring down latency to 30ms and improve throughput by 3x.

im a c# dev so im totally useless helping you to figure out what causes the bottleneck in rabbit because i dont know erlang. but there seems to be some kind of contention if the number of publishers increase.