msg_store_persistent usage suddenly spikes to ~100× actual queued messages in RabbitMQ 4.0.8 #13939
Description
Describe the bug
We’re running RabbitMQ 4.0.8 on Erlang/OTP 27.3.2. Over the last few weeks we’ve observed that msg_store_persistent (the on-disk store used by persistent queues) will at random times explode to 400 GiB+, even though the sum total of all messages in all queues at that moment is consistently around 3 GiB. This causes our broker node to rapidly fill its filesystem and eventually go into disk alarms or crash.
Queues setting:
x-queue-type:classic durable: true queue storage version: 2
Disk Usage:
`394G ./msg_store_persistent
256M ./msg_store_transient
3.9G ./queues
398G .
`
Reproduction steps
- Don't know flow, it happens randomly
Expected behavior
Ideally it should consume less disk space as there is not much messages.
Additional context
No response