RMQ running low on Memory

[BigBoulard]

Hi guys,

I'm working on a chat application that uses RMQ to route messages to rooms.

At startup, the application creates a single topic exchange (non-durable non-auto-deleted) and a pool of 20 connections.
When a user connects, the application:

• request one of the 20 connections from the pool
• creates a producer
• creates a consumer
• creates 2 channels: 1 for the producer and 1 for the consumer

side note: messages are transient and no ACK is required.

The design looks like this:

Then, I created a stress test script that creates pairs of users in a room and makes them chat indefinitely to see where and how it breaks.

After around 16 250 users, the application breaks with RMQ being low on memory. I can see this log on the RMQ instance:

[info] <0.354.0> vm_memory_high_watermark set. Memory used:3292753920 allowed:3290313523
[warning] <0.352.0> memory resource limit alarm set on node rabbit@74194ee8a9a3.
[warning] <0.352.0> 
[warning] <0.352.0> **********************************************************
[warning] <0.352.0> *** Publishers will be blocked until this alarm clears ***
[warning] <0.352.0> **********************************************************
[warning] <0.352.0> 
[info] <0.354.0> vm_memory_high_watermark clear. Memory used:3252813824 allowed:3290313523
[warning] <0.352.0> memory resource limit alarm cleared on node rabbit@74194ee8a9a3
[warning] <0.352.0> memory resource limit alarm cleared across the cluster
[info] <0.354.0> vm_memory_high_watermark set. Memory used:3327787008 allowed:3290313523
[warning] <0.352.0> memory resource limit alarm set on node rabbit@74194ee8a9a3.
...
...
[info] <0.354.0> vm_memory_high_watermark clear. Memory used:3284652032 allowed:3290313523
[warning] <0.352.0> memory resource limit alarm cleared on node rabbit@74194ee8a9a3
[warning] <0.352.0> memory resource limit alarm cleared across the cluster
[warning] <0.17632.10> closing AMQP connection <0.17632.10> (172.27.0.4:47786 -> 172.27.0.3:5673 - amqp_conn_18, vhost: '/', user: 'user'):
[warning] <0.17632.10> client unexpectedly closed TCP connection
[warning] <0.17536.10> closing AMQP connection <0.17536.10> (172.27.0.4:47674 -> 172.27.0.3:5673 - amqp_conn_6, vhost: '/', user: 'user'):
[warning] <0.17536.10> client unexpectedly closed TCP connection
[warning] <0.17608.10> closing AMQP connection <0.17608.10> (172.27.0.4:47762 -> 172.27.0.3:5673 - amqp_conn_15, vhost: '/', user: 'user'):
[warning] <0.17608.10> client unexpectedly closed TCP connection
[warning] <0.17600.10> closing AMQP connection <0.17600.10> (172.27.0.4:47750 -> 172.27.0.3:5673 - amqp_conn_14, vhost: '/', user: 'user'):
...
[warning] <0.17504.10> client unexpectedly closed TCP connection
[notice] <0.61.0> SIGTERM received - shutting down
[notice] <0.61.0> 
[warning] <0.497.0> HTTP listener registry could not find context rabbitmq_prometheus_tls
[warning] <0.497.0> HTTP listener registry could not find context rabbitmq_management_tls
[info] <0.230.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping unregistration.
[info] <0.627.0> stopped TCP listener on [::]:5673
[info] <0.21713.20> Closing all connections in vhost '/' on node 'rabbit@74194ee8a9a3' because the vhost is stopping
[info] <0.440.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@74194ee8a9a3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
[info] <0.440.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@74194ee8a9a3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
[info] <0.436.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@74194ee8a9a3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
[info] <0.436.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@74194ee8a9a3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
[info] <0.381.0> Management plugin: to stop collect_statistics.

It's the first time I use RMQ, so I'm not sure if the way I'm using it is okay and then either I'm doing something wrong or maybe I just need more resources ... or if this implementation is just non-sense.

[mkuratczyk]

What's the question here? Of course as you add more users / connections / channels, it will require more memory...
You can have a look at https://www.rabbitmq.com/memory-use.html for some hints on memory usage analysis, but no amount of analysis will change the fact that as you put more work on RabbitMQ, it uses more memory (but perhaps it could use less than it does).

Keep in mind that non-durable things (exchanges, queues) are pretty much going away soon (in 4.0 next year). You didn't say anything about the queues you use but the choices here may heavily impact the memory usage.

[BigBoulard]

Hi Michal,

Thank you for getting back to me so quickly.

I played with the durability and the auto-delete parameters of the queues for testing but now they are all durable and non-auto-deleted which means the app is responsible for deleting the queues. The queues are of the Classic type. I didn't make them Lazy cause it doesn't look like it's a good use case for that. They are not exclusive either.

I'm creating 16 500 users talking 1-to-1 in a chatroom, so there's:

• 20 connections (cause I think 1 connection can handle 2 048 channels)
• 1 topic exchange
• 33 000 channels (1 user = 1 channel for the producer and 1 channel for the consumer)
• 16 500 queues (1 per chatroom)

About channels, the test if about 1-to-1 chatroom but if a user participates in X chatroom, he will have X consumers and X channels (I can't see how I can share a channel between 2 consumers and call ch.Consume(...) on both).

I wanted to keep track of the UUID and Session ID of the user who produce a message, so the routing key used by the producer on the topic exchange is msg.UUID.SID.ROOM_ID which makes a very long key now that I'm talking about it ....

Then the users from a chatroom have their consumer listening on this routing key: msg.*.*.ROOM_ID

So in the end, I have this:

# Producer Routing Key
msg.UUID.SID.ROOM_ID`
# Consumer Routing Key
msg.*.*.ROOM_ID
# Queue name
UUID.SID.ROOM_IDqueue

I wanted to check if the numbers were way too high compared to what you can see in similar cases, or if you could point out an obvious implementation error that I may have overlooked (queue names? use something else than a topic exchange? MQTT or another protocol than AMQP, etc.).

I've made another test where I just keep track of the WebSockets in a map of room IDs.
Obviously, it opens other challenges, like:

• Scaling out cause I would need to load balance the WebSockets etc.
• Offline delivery but here messages are already stored in a database cluster, and the last X messages can be fetched upon user connection. If I need to notify an offline user, I think I can check the online status in DB and just send an email as well.
• probably other challenges I don't have in mind already

But putting that aside, on a single server, it stands 90,000 users which makes a huge difference, and I just want to be convinced that RabbitMQ is the right choice here and that I'm doing something wrong.

[mkuratczyk]

In 3.12+, all queues are effectively lazy: https://blog.rabbitmq.com/posts/2023/05/rabbitmq-3.12-performance-improvements/#classic-queues-changes-to-the-lazy-mode

If you are not using 3.12, you should. This alone may lower your memory usage. And while you're at it, you can set x-queue-version=2.

However, you didn't share anything about what is using the memory currently, so we have not data to even try to help.

I have never built a chat app, so I don't have a ready to use answer for you. There are many knobs you can try to use to lower the memory usage (eg. https://www.rabbitmq.com/networking.html#tuning-for-large-number-of-connections-tcp-buffer-size). There's also an option to use MQTT which uses much less memory per connection in 3.12 (https://blog.rabbitmq.com/posts/2023/03/native-mqtt/) and has "no queue" queues for QoS0 (messages are delivered directly to the consumer, skipping any actual queue). Whether it's a good option for your app - that's not my place to say.

[BigBoulard]

Thank you for getting back to me so quickly :)
I'm using the rabbitmq-management image which is 3.12.4, now I know that "classic" is "lazy"
I'll dig into the docs and try to see what causes the relatively high memory consumption but my bet is about the queues, so I'll take a look at MQTT and this go client github.com/eclipse/paho.golang as well. Don't hesitate if you know someone with feedback on this ... I also heard about the STOMP protocol but it sounds to be a little bit rough (maybe not the right term here)

[mkuratczyk]

The protocol is one thing, the implementation is another. Historically MQTT and STOMP used even more resources than AMQP 0.9.1 in RabbitMQ, not because of the protocols themselves but how they were implemented years ago (as proxies to internal AMQP0.9.1 connections). 3.12 brought a complete redesign of MQTT support - the "native" meaning it no longer proxies to AMQP 0.9.1 internally. There's work in progress to have native AMQP 1.0 and native STOMP support as well, so they will use less memory as well.

[BigBoulard]

@mkuratczyk I think the problem is that I'm just using too many queues. I'm using 2 queues per user and it's too much.

I'm sure that Rabbitmq could still be a good help once I put a layer 3 load balancer in front of some app nodes. In this configuration, a user web socket would be always assigned to the same node. Users' web sockets would be shared across the nodes then.

In case a message is sent to a chat room, a single instance can still send the message to the users it manages ... but how do I know if there are other instances managing some users of the same chatroom and how to make them receive and send the message too?

It reminds me of Cassandra-like clusters where data is stored across nodes based on the partition key, with the coordinator node in charge of contacting the good node and getting the response.

Here it's a bit different though, but maybe each node could have a producer and a consumer or something like this ... If you have an idea, or know a common pattern involving RMQ for this case, that would be gold to me.