RabbitMQ disk fills up with segment files

[4danmi]

Hi,
We had a similar issue to #9853. (We are still using the same settings).
When we executed rabbitmq-queues quorum_status node3 raft_state was stuck on await_condition forever. Then, we used rabbitmq-queues delete_member + add_member to return the node to the queue. We also tried to restart the consumers, but it didn't work.

It seems both the leader (node2) and node3 snapshot indexes are stuck on 1,577,642,039 while the node1 snapshot index is stuck on 1,620,936,426. The commit index is the same between all the nodes and continues to grow. (The queue itself works fine).

Unfortunately, the cluster is on a closed network, so we can't export the segment files. If necessary, we can copy the config file manually.

[michaelklishin]

@4danmi sorry but we cannot help you with the amount of information you provide. We do not even know what version is used.

If a quorum queue does not delete its segment files, the most likely factor is that a consumer that has consumed a message is stuck and the oldest messages delivered remains unconsumed. When this happens, quorum queue Raft log will keep growing without deleting any data.

In fact, in #9853 another team member has provided a hypothesis and you haven't responded to that:

Most likely something like a faulty consumer held a lock on a message and
eventually released the lock. You didn’t really provide any information for
us to use to even provide a guess with

Filing issues repeatedly without new relevant information won't get you anywhere.

[michaelklishin]

Default consumer delivery timeout was introduced exactly because of this scenario. Lower it to, say, 5m (that should be plenty of time for the absolute majority of consumers) and compare the outcome.

Removing and re-adding replicas has the effect of removing/canceling consumers, which in turn can requeue the "problematic" (unconfirmed) message.

We are not aware of other scenarios where in a recent version of RabbitMQ quorum queues would leave segment files around. It's almost always a consume that consumes and never acknolwedges.