RabbitMQ concurrent active connections stuck at around 129K

[rajatgl17]

Data-flow overview: Mobile application (millions of users) publish click stream events data to our RabbitMQ cluster (behind a AWS network load balancer) with MQTT protocol. We have a cluster of RabbitMQ with 13 nodes (c7.4xlarge). From AMQP topic, data is moved to a random exchange which distributes data to 13 different quorum queues. And from these 13 quorum queues, an application consume data with AMQP 0-9-1 protocol.

Problem: We are load testing our infrastructure with a distributed JMeter as of now. But with all the setup we are stuck at around 129K concurrent connections, although we have configured 250K concurrent connections in JMeter.

What we have already checked:

1. JMeter

• Infra/Hardware wise there is no shortage of any resource CPU, RAM, ulimit etc.
• We even tried horizontally and vertically scaling JMeter nodes, even with lesser nodes, still the concurrent connections stuck at 129 K

2. RabbitMQ

• Infra/Hardware wise there is no shortage of any resource CPU, RAM, disk space, Network bandwidth, Disk IOPS, ulimit etc.
• We even tried horizontally (9 nodes, 13 nodes, 15 nodes) and vertically (c7.8xlarge) scaling RabbitMQ nodes, even with lesser nodes (9 nodes), for all the combinations the concurrent connections stuck at 129 K
• We are using RabbitMQ version 3.13.3
• Followed guidelines at - https://www.rabbitmq.com/docs/networking#tuning-for-large-number-of-connections

[mkuratczyk]

Check out the setup we had for the tests: https://github.com/rabbitmq/mqtt-testing/tree/main/many-conn
You will probably find some hints about tuning for many connections and in particular, ERL_MAX_PORTS might be what you're missing.

[rajatgl17]

Hi @mkuratczyk,
We've already increased the value of this parameter significantly, but the problem persists.

[rajatgl17]

Agreed @mkuratczyk, I need to put more details.

Can you please recheck the link that you have shared. Either it's broken or is of a private repo.

There was no impact in any metric when we added more nodes to the cluster and/or changed the parameter you shared. Log level was at info and at RabbitMQ side there were no error logs.

On publisher side we can see logs saying unable to connect, timeout exception.

[mkuratczyk]

Sorry, I didn't notice it's not a public repo. We can look into making it public later on, but for the time being, here's the RabbitMQ Kubernetes deployment we used there. Of course it does mean you should copy every setting, but it should give you an idea of what settings we had to tweak.
rabbitmq.yaml.txt

[rajatgl17]

Hi @mkuratczyk
Even after using the same set properties:

We are still stuck at 129K connections

No error logs in RabbitMQ:

Error at Jmeter side (timeout exception):

[ansd]

https://github.com/ansd/rabbitmq-mqtt/tree/main/background-connections demonstrates how to scale to 1 million MQTT connections.

[ansd]

For example management metrics are switched off as you can see in https://github.com/ansd/rabbitmq-mqtt/blob/2f012001b598854e8b36459be2397e879d0c9020/background-connections/mqtt-rabbit.yml#L27 and as is explained in the blog post in https://www.rabbitmq.com/blog/2023/03/21/native-mqtt#memory-usage
Your screenshot shows that you didn't switch off management metrics.

Follow the tips in https://www.rabbitmq.com/docs/mqtt#scalability
At what rate do MQTT clients connect to RabbitMQ?
More RabbitMQ nodes does not necessarily mean faster connection establishment. The opposite can be true due to Mnesia. You can try using fewer RabbitMQ nodes or test with Khepri.