RabbitMQ Memory Optimization Help

[dormanze]

Hello, everybody.
rabbitmq version: 3.11.9
erlang version: 25.22
I have two RabbitMQ instances, which carry about 65000 queues, among which about 5,000 are persistent queues, and others are automatically deleted exclusive queues. The size of a single message is less than 5 KB. The following figure shows the memory usage.

I have the following questions:
1:Why does the plug occupy 4.3 GB memory? I have only enabled rabbitmq_web_dispatch, rabbitmq_management, and rabbitmq_management_agent. Why do these plug-ins occupy so much memory?
2:I also found that after I restarted the two instances, only the queue and the consumer was initialized memory was not high. The instance with the highest memory usage was about 6 GB. After a period of message production and consumption, the memory keeps rising until it reaches the current value.I'm a little suspicious of a memory leak.
3:Based on the current usage, do you have any suggestions on modifying the configuration to optimize the current memory usage?

[mkuratczyk]

Since you have a lot of queues and channels, the plugin that uses a lot of memory is almost certainly the management plugin, since it collects all those metrics that you are showing in the screenshot. We recommend, especially in environments with a lot of queues/connections/channels, using the Prometheus endpoint for monitoring and turning off the Management metrics. This way you don't store metrics within RabbitMQ, but externally (but you also won't get these graphs in the Management UI). If you are not ready for that, you can play with https://www.rabbitmq.com/management.html#sample-retention to limit the amount of data collected/stored.

Second point is the number of channels you have. Given you have ~300 connections, I'm don't understand why you need ~14000 channels. Perhaps instead of that many channels that declare that many queues, you can just have fewer queues bound with multiple routing keys so that the same apps receive the same messages, but using much fewer channels and queues? That should lower the memory usage significantly.

[mkuratczyk]

rabbitmq_management_agent and rabbitmq_prometheus are independent of each other, so yes, enabling the Prometheus plugin doesn't automatically reduce the amount of data stored by the management plugin.

The best thing you can do is to lower the number of channels and queues you use. I don't see a good reason to have that many, if you have just ~300 connections to consume them.

As for the API - we recently made improvements to /api/queues, mostly focused on response time, but I guess this could affect memory usage as well. This particular change will be included in 3.13. Can you share what API endpoints you query and how (a full URI string with the filtering/pagination parameters)? It may or may not be related to your memory usage but either way would help us understand how the API is used and how we can make it more performant in future versions.

[dormanze]

I'm very grateful for your answer.
The requests I often use are as follows:

/api/health/checks/local-alarms
/api/vhosts
/api/nodes
/api/queues?page=1
/api/overview
/api/nodes/rabbit@adc-rabbitmqservice-0
/api/nodes
/api/health/checks/port-listener/rabbit@adc-rabbitmqservice-0

I also often view some queue information on the management interface.
Does the community consider not caching metric information, but using each query, providing the latest information each time? Because I've found that in the worst case, the cached information for metrics can take up half of the memory used, which is hard to accept.

[mkuratczyk]

I'm not sure I understand the question but if I do - that's basically why we suggest using Prometheus instead. We officially deprecated the Management stats 2+ years ago: https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#disable-metrics-delivery-via-the-management-api--ui. However, many users love the built-in nature of the Management plugins/stats and it is sufficient for small deployments, so it's a hard to remove feature. We've not decided what exactly we'll do about it - perhaps changing the defaults to lower retention / granularity by default or something like that for now. But for large deployments - just disable Management stats and use prometheus (or at least set a very short retention period for the management stats)

[dormanze]

Thank you again for your detailed answer.
I tried to switch to the prometheus plug, and sure enough, the memory usage was about the same as the management plug, but the advantage was that the response was much faster. I will continue to adjust the metric collection period and cache.

[dormanze]

I finally decided to use the parameter management_agent.disable_metrics_collector: https://www.rabbitmq.com/management.html#disable-stats and turn on the rabbitmq_prometheus plugin.