Continuous increase in memory usage of other_ets

[fcovatti]

We have been experiencing a continuous increase on the memory usage of the other_ets table that can lead to an OOM issue. We have a cluster with 3 nodes and we have to perfomr every 2 to 3 months a restart as it reachs 50% memory usage we have available on the server, then memory is freed. What could be causing the other_ets memory increase? We have been using rabbitmq alongside 5 celery instances and around 80 mqtt connections.

We have tried updating the rabbitmq to newer version from 3.8.12 to 3.11.9 but that did not help.

Logs of memory usage at this moment:

reserved_unallocated: 0.0825 gb (35.23%)
other_ets: 0.0505 gb (21.56%)
code: 0.0385 gb (16.46%)
other_proc: 0.0189 gb (8.08%)
other_system: 0.0186 gb (7.95%)
plugins: 0.0066 gb (2.84%)
connection_other: 0.0048 gb (2.07%)
mgmt_db: 0.0033 gb (1.4%)
binary: 0.0028 gb (1.19%)
allocated_unused: 0.0023 gb (0.99%)
atom: 0.0015 gb (0.65%)
queue_procs: 0.0013 gb (0.55%)
metrics: 0.0013 gb (0.54%)
connection_channels: 0.0003 gb (0.14%)
msg_index: 0.0003 gb (0.14%)
connection_readers: 0.0002 gb (0.1%)
mnesia: 0.0002 gb (0.1%)
connection_writers: 0.0 gb (0.02%)
quorum_ets: 0.0 gb (0.01%)
quorum_queue_procs: 0.0 gb (0.0%)
quorum_queue_dlx_procs: 0.0 gb (0.0%)
stream_queue_procs: 0.0 gb (0.0%)
stream_queue_replica_reader_procs: 0.0 gb (0.0%)
queue_slave_procs: 0.0 gb (0.0%)
stream_queue_coordinator_procs: 0.0 gb (0.0%)

rabbitmq-plugins list
Listing plugins with pattern ".*" ...
 Configured: E = explicitly enabled; e = implicitly enabled
 | Status: * = running on [email protected]
 |/
[  ] rabbitmq_amqp1_0                  3.11.9
[  ] rabbitmq_auth_backend_cache       3.11.9
[  ] rabbitmq_auth_backend_http        3.11.9
[  ] rabbitmq_auth_backend_ldap        3.11.9
[E*] rabbitmq_auth_backend_oauth2      3.11.9
[  ] rabbitmq_auth_mechanism_ssl       3.11.9
[  ] rabbitmq_consistent_hash_exchange 3.11.9
[  ] rabbitmq_event_exchange           3.11.9
[e*] rabbitmq_federation               3.11.9
[E*] rabbitmq_federation_management    3.11.9
[  ] rabbitmq_jms_topic_exchange       3.11.9
[E*] rabbitmq_management               3.11.9
[e*] rabbitmq_management_agent         3.11.9
[E*] rabbitmq_mqtt                     3.11.9
[  ] rabbitmq_peer_discovery_aws       3.11.9
[e*] rabbitmq_peer_discovery_common    3.11.9
[  ] rabbitmq_peer_discovery_consul    3.11.9
[  ] rabbitmq_peer_discovery_etcd      3.11.9
[E*] rabbitmq_peer_discovery_k8s       3.11.9
[E*] rabbitmq_prometheus               3.11.9
[  ] rabbitmq_random_exchange          3.11.9
[  ] rabbitmq_recent_history_exchange  3.11.9
[  ] rabbitmq_sharding                 3.11.9
[  ] rabbitmq_shovel                   3.11.9
[  ] rabbitmq_shovel_management        3.11.9
[E*] rabbitmq_stomp                    3.11.9
[  ] rabbitmq_stream                   3.11.9
[  ] rabbitmq_stream_management        3.11.9
[  ] rabbitmq_top                      3.11.9
[  ] rabbitmq_tracing                  3.11.9
[  ] rabbitmq_trust_store              3.11.9
[e*] rabbitmq_web_dispatch             3.11.9
[E*] rabbitmq_web_mqtt                 3.11.9
[  ] rabbitmq_web_mqtt_examples        3.11.9
[  ] rabbitmq_web_stomp                3.11.9
[  ] rabbitmq_web_stomp_examples       3.11.9

Logs of memory usage on previous version 3.8.12 (note that quorum_ets also included other_ets in the memory report on previous version I guess) :

quorum_ets: 0.2352 gb (55.36%)
other_proc: 0.0494 gb (11.64%)
allocated_unused: 0.0432 gb (10.18%)
code: 0.0356 gb (8.39%)
other_system: 0.0291 gb (6.86%)
plugins: 0.0095 gb (2.25%)
other_ets: 0.0077 gb (1.81%)
connection_other: 0.0052 gb (1.23%)
binary: 0.0036 gb (0.84%)
mgmt_db: 0.0017 gb (0.4%)
atom: 0.0015 gb (0.36%)
queue_procs: 0.0008 gb (0.19%)
metrics: 0.0005 gb (0.11%)
connection_readers: 0.0005 gb (0.11%)
quorum_queue_procs: 0.0004 gb (0.1%)
msg_index: 0.0003 gb (0.08%)
connection_channels: 0.0002 gb (0.06%)
mnesia: 0.0002 gb (0.04%)
connection_writers: 0.0001 gb (0.01%)
queue_slave_procs: 0.0 gb (0.0%)
reserved_unallocated: 0.0 gb (0.0%)

My suspicion would be the reconnect of the mqtt being the cause of the problem, as we do not have not fully stable connections and reconnections happen from time to time. We have also noticed that when a node was reconnecting every 1 second, that caused the memory consumption to increase at a faster rate. Is there anything we can do to prevent the increase in memory usage due to the reconnections?

[mkuratczyk]

Please start by upgrading to a supported version. Lots of issues have been fixed since 3.8/3.9.

[mkuratczyk]

How much memory did you assign to RabbitMQ? Based on quorum_ets: 0.2352 gb (55.36%), it seems like you have 0.5GB? If so, I guess you are hitting: https://www.rabbitmq.com/quorum-queues.html#resource-use

[fcovatti]

Usually 2GB is allocated to RabbitMQ. We are not using quorum queues (only classical queues), interestingly the rate of change we experience is much slower than the one on the graph in resources usage. On the graph below the valleys are when we restarted the cluster.

Alos, I was going over the doc and notice this setting https://github.com/rabbitmq/rabbitmq-website/blob/3263c6f16d24efc21d29da2a9092a966cd9bee30/site/mqtt.md#memory-footprint-of-raft-log-memory-tables

I will try to setup a lower value for the Raft Log Memory maximum size and give you some feedback in here.

[mkuratczyk]

The exact memory usage pattern, even for quorum queues, depends on the publish rate - how quickly the WAL gets full, at which point it's flushed. The docs talk about the pattern, not how often those ups and downs will occur, since we can't know that.

RabbitMQ 3.12 will have a massively reduced memory usage for MQTT connections (a blog post coming soon) so maybe this will help you as well. However, restarting a server because it uses 700MB of memory seems strange to me to be honest, unless this is an edge device. For classic queues, you can also try lazy mode or classic queues v2. Especially in 3.12, classic queues v2 will provide lower memory usage in many cases (yep, lots of goodies coming soon).

[fcovatti]

Hi @mkuratczyk just to confirm that we have seen this sawtooth behavior of the memory in our servers and as we mainly use MQTT (no quorum queues) we proceed by setting the raft.wal_max_size_bytes = 64000000. We can see that the memory peaks and valleys are in the range of 150MB of memory. Also to test this we keep one MQTT client making connections every second which results in an interval between peaks in the range of 2 to 3 days. Many thanks for all the help and clarification, it helped me understand better and troubleshoot what we were seeing as a potential issue as just a false positive alert.

[mkuratczyk]

Thanks for reporting back. FYI, the RAFT-based component of our MQTT plugin (described in https://www.rabbitmq.com/mqtt.html#consensus) goes away completely in 3.12. The blog post describes all kinds of improvements and mentions this as well (search for delete_ra_cluster_mqtt_node): https://blog.rabbitmq.com/posts/2023/03/native-mqtt/