Sudden increase in the MEMORY of RabbitMQ Cluster POD..and It comes down on restart of the POD

[ankitgr8]

Below is the image from our setup where suddenly the Memory for two POD goes UP and it came down only when we restarted the POD..

During this time there were lot AMQP Error connections, and we believe that due to this error connection.. there was huge Channel and connection churn happened which may be the reason for the low memory.. We have seen this only for rabbitmq-cluster-0 and rabbitmq-cluster-1 nodes only.. The memory breakdown does show the memory consumption in "other_proc" which is pointing to the internal rabbitmq_event being generated due to channel/connection churn

What we are looking is to know the reason of why the connections were getting disconnected.

Attached are the logs for all the 3 Nodes of a RabbitMQ Cluster..
rabbitmq-cluster-0.tar.gz
rabbitmq-cluster-0-1.tar.gz
rabbitmq-cluster-1.tar.gz
rabbitmq-cluster-2.tar.gz

[mkuratczyk]

You have ~10000 quorum queues which is already stretching it and judging by their names, you probably have a high quorum queue churn as well? Combined with connection churn, most likely you are overloading the Erlang distribution links (communication within the cluster). Step one is probably to reduce this load, probably by reconsidering your topology (number and type of queues) and preventing connection churn.

[ankitgr8]

The queues are static.. not dynamic.. they do not delete.. so there is no queue churn... The name of the queue has UUID.. but they are static in nature...
So why connection churn is so high is I am looking for help

[mkuratczyk]

Ok. Either way, you have timeouts - I can't tell you why, but likely due to the erlang distribution links being overloaded (check the Erlang Distribution dashboard). Or you may just have a unreliable network connection between the nodes.

It's not clear how you made the transition from "other_proc" to "rabbit_event". It might be true, but please explain why you suspect rabbit_event.

10k quorum queues is a lot. Quorum queues, even idle, perform some cross-cluster housekeeping operations. If you publish/consume a significant amount of messages on top of that, especially if the messages are large, you may simply overload the cluster or the links.

You may also have a look at rabbitmq-diagnostics observer (sorted by memory/reductions/mailbox) to get an idea of what the cluster is busy doing. A more involved method which may also be helpful is profiling: https://www.rabbitmq.com/blog/2022/05/31/flame-graphs.

My previous advice still stands: ask yourself whether you need 10k quorum queues in the first place. And make sure your network is reliable between the nodes.

Timeouts on the queue level (visible in the logs) may lead to clients being disconnected, which may lead to high conneciton churn if they quickly reconnect, potentially exacerbating the initial problem.

[lukebakken]

I see a lot of timeout errors in your logs:

2024-03-06 12:13:02.804185+00:00 [error] <0.18122.1893> Error on AMQP connection <0.18122.1893> (10.206.161.122:54105 -> 10.206.161.121:5671 - dp.connection.factory#31a4db4f:1, vhost: '/', user: 'rabbitmq', state: running), channel 405:error while terminating:
2024-03-06 12:13:02.804185+00:00 [error] <0.18122.1893> {{badmatch,
2024-03-06 12:13:02.804185+00:00 [error] <0.18122.1893>      {timeout,
2024-03-06 12:13:02.804185+00:00 [error] <0.18122.1893>          {'%2F_gms.request.dp.job.6254d39a-5067-48dd-9b52-f2e5c4977589.q',
2024-03-06 12:13:02.804185+00:00 [error] <0.18122.1893>              '[email protected]'}}},
2024-03-06 12:13:02.804185+00:00 [error] <0.18122.1893>  [{rabbit_quorum_queue,consume,3,
2024-03-06 12:13:02.804185+00:00 [error] <0.18122.1893>       [{file,"rabbit_quorum_queue.erl"},{line,804}]},

The above happened when a consumer tried to start. My guess is that your cluster is overloaded.

@ankitgr8 if you have a support contract with Broadcom for RabbitMQ, please use the official support channels.

cc @kjnilsson

[ankitgr8]

It's not clear how you made the transition from "other_proc" to "rabbit_event". It might be true, but please explain why you suspect rabbit_event
[Ankit] Due to the connection churn, the internal events which are generated resulting in high rabbit_event..

We had 3 node cluster.. and there was 1:1 mapping of POD and NODE, each POD is part of one dedicated node.. With this configuration.. we are seeing issue as mentioned above... after moving all PODS in a single NODE.. we did not encounter any issue..

So what I am not clear is the ERLANG inter node(in this case inter POD.. since all the PODS are in same NODE..) is still happening and all other configuration and load remains the same, why we do not see any issue..

Once we move PODS accross NODE, what ERLANG communication changes?...Does "Inter-node Communication Buffer Size" has any role to play... since I can see these warning in my logs -- "rabbit_sysmon_handler busy_dist_port"
And does this can also leads to connection disconnects and also impact any quorum queue communication at RAFT level

[mkuratczyk]

Communication over the loopback interface is magic. It has little to do with a real network: it's much faster (much lower latency, much higher throughput), default TCP buffers are completely different on the loopback interface (you might want to tune them on your real interface - this might help, although certainly won't make a real network as fast as loopback). Yes, "Inter-node Communication Buffer Size" is one of the things you can try to tune.

[ankitgr8]

And also does Streams will perform better with quorum queues.. with such a scale (no of queues).. we are aiming to have 100k quorum queues.

[mkuratczyk]

For the third time: most likely you don't actually need that many queues/streams. Rather than saying "I need 100K", think about alternative designs. You can share your requirements here and your current solution with lots of queues and we might be able to suggest an alternative design (maybe 1 stream? maybe a superstream, maybe a handful of queues and consistent hash exchange and so on).

[ankitgr8]

At a high level the requirements are-- We have device which report data.. the count of these device can go up to 20k.

1. Each device reports 2 category of data...out of which 1 category of data needs to be ingested real time.(No latency with respect to the messages spend time in queue). Another category of data can have some latency.
2. Ordering of messages is Important for both the category of data
3. All the devices are sending the data in parallel
4. Scale of messages -- category 1 data can be report at a scale of 1-2 messages per second (across all devices -- this worst case scenario).. where as category 2 data can be 100 messages per 3 hours (accross all devices)
5. Size of category 1 data will be arround 10-50 KB where as size of category 2 data can be between 500 KB to 1 MB

[ankitgr8]

@mkuratczyk .. Shared High level requirements.. Any guidance on the pattern we should follow in rabbitMQ...
One thing I want to mention that we are running with micro service architecture,, and we have 3 services which are used in data processing of data which is pushed from these devices and every Service Have 2 queues created for each device.. so if we say we want to support 20k device which means 20k x 3(service) x 2(queue per device) = 120k queues..

[mkuratczyk]

Why do you publish to a different queue from each device? A queue is a consumer-side concern, not really a publisher-side one. I see no reason to create a queue per device, let alone a few queues per device, to facilitate fan-in scenario (from many devices to a few consumers; it would make sense for communication in the opposite direction - when all devices need to receive a message sent from some central application). Publishing all of these messages to a single queue could be an option (well, probably 1 queue for urgent messages and another one for the other type). Distributing between a few queues could be an option. A consistent has exchange could be an option.

[ankitgr8]

@mkuratczyk Thanks for the reply.

There is no issue for publisher, publisher can send data to single queue. The issue is at the consumer end only...

1. Message need to be processed in order.. hence I cannot have multiple consumer listening to same queue
2. All devices publishing message to single queue with 1 message per sec rate and considering 20k devices result in 20k messages per sec., would lead to huge latency .. considering a message take 10 m/s to process would result in processing only 100 message per sec as compare to 20k incoming message...Which means we need at least 200 queue to process 20k message in real time.

So as per your suggestion we can use consistent hash exchange which can send message to 200 queues.. and all the 20k devices messages are distributes among these 200 queues...

The Only issue we have is.. if some messages in queue take time to process. they impact the processing of messages from another device. if both are sending message to same queue.----This was one of the reason of segregating the queues per device.

[mkuratczyk]

well, as you see, your approach is not leading to smooth processing either... I think it'd be much easier to accomplish what you need if you don't have tens of thousands of queues and connections to deal with. The cluster will have more resources, you can use more than 200 queues so that the msg/queue ratio is lower (but a few times more, not thousands of times more).

You can consider streams, super streams, stream filtering. There are options. 100K+ QQs or streams is not an option currently.