Rabbit producer connection throtting

[sandeepkr977]

Hi,

We are using Rabbit (RabbitMQ 3.13.2 Erlang 26.2.5) in production environment. Since, long time we are facing producers connection throttling issue. Whenever data producing speed increases or consumers joins/drops, producers connection become throttle.

Currently, we are producing data at speed 20K TPS and 1 message size is 100KB. We are running our system with 3 producer (1 channel each). Each Rabbit MQ node have 64GB and 1 TB disk. At the time of throttle issue, RAM and disk is completely empty. There is no lag in queues as well.

Is there any way to resolve producer connection throttle issue?

[mkuratczyk]

Can you reproduce this problem using https://perftest.rabbitmq.com/? If yes, please share the arguments used to simulate your workload. This would give us a way to reproduce and all the other details about the workload (you didn't even mention the queue type). Also, what do you mean by "There is no lag in queues as well"?

100kb is a pretty large message, so "best bang for your buck" tuning tip I can give you is to reduce the message size if possible. For example, compress it before publishing (and of course decompress after consuming).

TCP buffers also play a huge role with such messages: https://www.rabbitmq.com/docs/runtime#distribution-buffer

[sandeepkr977]

Queue type: Quorum
There is no lag in queues as well -> There is no data in ready state. Everything is consuming on time.

Test Cases Using PrefTest :

java -jar -Xms6g -Xmx6g perf-test.jar --producers 1 -X 1 --consumers 0 --rate 1000 --size 102400 --time 246400 --queue performance-test-95-0 --routing-key performance-test-95-0 --exchange test-exchange --uris 'amqp://user:[email protected]:5672/v-host,amqp://user:[email protected]:5672/v-host,amqp://user:[email protected]:5672/v-host’ --quorum-queue

 
java -jar -Xms6g -Xmx6g perf-test.jar --producers 0 --consumers 360 --size 102400 --time 246400 --queue performance-test-95-0 --routing-key performance-test-95-0 --exchange test-exchange --uris 'amqp://user:[email protected]:5672/v-host,amqp://user:[email protected]:5672/v-host,amqp://user:[email protected]:5672/v-host’ -R 10 --quorum-queue

[mkuratczyk]

Using these commands I seem something like this from the publisher when the consumers join a bit later:

id: test-155446-528, time 24.000 s, sent: 1000 msg/s
id: test-155446-528, time 25.000 s, sent: 1001 msg/s
id: test-155446-528, time 26.001 s, sent: 899 msg/s
id: test-155446-528, time 27.000 s, sent: 395 msg/s
id: test-155446-528, time 28.000 s, sent: 1707 msg/s
id: test-155446-528, time 29.000 s, sent: 1001 msg/s
id: test-155446-528, time 30.000 s, sent: 1001 msg/s

So about 2 seconds drop in throughput while the consumers catch up and then everything stabilises. Is this what you are seeing or do you see a longer or deeper drop in publisher throughput?

Since this is a single queue scenario, the amount of cores and RAM doesn't matter much since everything gets "serialized" at the queue layer. A single queue can't use multiple cores at the same time.

[mkuratczyk]

After playing around with this scenario more and talking to a colleague, we believe the main problem is the prefetch size. perf-test uses a large prefetch by default which means when you connect the consumers, especially as many as 360, the queue has to read and dispatch a lot of messages. If you set the prefetch to 1 or 2, you should not see such a drop or it should be much less severe.