Growing number of QQ segment files and disk footprint

[anaskhantpl]

Issue Details:

Cluster Configuration: 3 nodes
OS: Ubuntu Server 22.04 x86_64
RabbitMQ Version: 3.13.1
Symptoms: Segment files in quorum queues grow rapidly, consuming approximately 1GB every 5 minutes or less.
Logs: No related errors or warnings found in the logs.
Configuration: See attached rabbitmq.conf.

Description:

We are running a 3-node RabbitMQ v3.13.1 cluster on Ubuntu Server 22.04 x86_64. The cluster has been operating without issues for several weeks. However, we have recently encountered a problem where segment files in quorum queues grow rapidly, consuming disk space at an alarming rate (approximately 1GB every 5 minutes or less).

Despite the rapid growth of segment files, the logs do not show any related errors or warnings. The system appears to be running normally. Here are some excerpts from the logs:

tail -f /var/log/rabbitmq/[email protected]
2024-08-06 18:42:05.498126+02:00 [info] <0.968250.0> connection <0.968250.0> (192.162.2.22:44564-> 192.168.1.2:2343): user 'x_user' authenticated and granted access to vhost '/'
2024-08-06 18:42:08.655364+02:00 [info] <0.968268.0> accepting AMQP connection <0.968268.0> (138..2.167: -> 192.162.2.22:44564)
2024-08-06 18:42:08.838129+02:00 [info] <0.968268.0> connection <0.968268.0> (192.162.2.22:44564 -> 192.168.1.2:2343): user 'x_user' authenticated and granted access to vhost '/'
2024-08-06 18:42:09.228281+02:00 [info] <0.968268.0> closing AMQP connection <0.968268.0> (192.162.2.22:44564 -> 192.168.1.2:2343, vhost: '/', user: 'x_user')
2024-08-06 18:42:21.466256+02:00 [info] <0.968539.0> accepting AMQP connection <0.968539.0> (192.168.1.1:4404 -> 192.168.1.2:2343)
2024-08-06 18:42:21.649098+02:00 [info] <0.968539.0> connection <0.968539.0> (192.168.1.1:4404 -> 192.168.1.2:2343): user 'x_user' authenticated and granted access to vhost '/'
2024-08-06 18:42:22.014199+02:00 [info] <0.968539.0> closing AMQP connection <0.968539.0> (192.168.1.1:4404 -> 192.168.1.2:2343, vhost: '/', user: 'x_user')
2024-08-06 18:42:24.527116+02:00 [info] <0.690.0> queue 'campaignchunk' in vhost '/': detected a new leader {'%2F_campaignchunk',rabbit@rm1} in term 57
2024-08-06 18:42:25.363211+02:00 [info] <0.968569.0> accepting AMQP connection <0.968569.0> (192.162.2.22:44564 -> 192.168.1.2:2343)
2024-08-06 18:42:25.550186+02:00 [info] <0.968569.0> connection <0.968569.0> (192.162.2.22:44564 -> 192.168.1.2:2343): user 'x_user' authenticated and granted access to vhost '/'

Configuration File:
Here is the rabbitmq.conf file we are using:

cluster_formation.peer_discovery_backend = classic_config

cluster_formation.classic_config.nodes.1 = [email protected]
cluster_formation.classic_config.nodes.2 = [email protected]
cluster_formation.classic_config.nodes.3 = [email protected]

vm_memory_high_watermark.relative = 0.7
disk_free_limit.absolute = 4G

Enabled Plugins:

# cat /etc/rabbitmq/enabled_plugins 
[rabbitmq_management,rabbitmq_prometheus].

Reproduction steps

NILL

Expected behavior

We seek assistance in identifying the root cause of this issue and any potential configuration changes or bug fixes that might resolve it. Thanks!

Additional context

No response

[lukebakken]

https://github.com/rabbitmq/rabbitmq-server/blob/main/COMMUNITY_SUPPORT.md#who-is-eligible-for-community-support

If you would like assistance from Team RabbitMQ, please start by using the latest version of RabbitMQ (3.13.6) to reproduce this issue.

In addition, a very small excerpt from your logs is nearly useless. Please archive your FULL log files (after upgrading RabbitMQ) and attach them to this discussion if you experience this issue again.

The cluster has been operating without issues for several weeks. However, we have recently encountered a problem where segment files in quorum queues grow rapidly,

If your environment ran normally for weeks, then this issue started, what changed? If you have historical monitoring information for this cluster (you DO monitor it, correct?), please check to see if your workload has changed. Have you deployed new client applications, or any updates to them?

[michaelklishin]

There was at least one similar issue resolved in 3.13.x, please upgrade to 3.13.6. We will not provide any assistance without a set of steps to reproduce and against an outdated patch version.

[michaelklishin]

However, we have recently encountered a problem where segment files in quorum queues grow rapidly,
consuming disk space at an alarming rate (approximately 1GB every 5 minutes or less)

Queues that have fast publishers or large messages and no consumers will result in runaway disk space usage. It should be fairly obvious why.

Inspecting message counts, including unacknowledged messages, across all queues in all virtual hosts would be one obvious way to confirm this hypothesis.

Introducing guard rails such as queue length limits is a good idea.

[michaelklishin]

@mkuratczyk has confirmed that 3.13.6 includes potentially (we do not have any reproduction steps or relevant details such as node data directories) relevant changes.

[kjnilsson]

Do each rabbit node show the same growth profile? Can you restart any consuming applications for that queue and see if it helps?