quorum queue with no consumers and TTL can have continually increasing segment file size

[chadknutson]

I'm running RabbitMQ 3.12.10 with Erlang 26.0.2 on macosx. The quorum queue set up is as follows:

• q1 (quorum queue) has a consumer that always rejects messages (my consumer waits 3 seconds before rejecting) with requeue set to true
• message dead lettered to q2 (quorum queue), which has a message TTL of 10 seconds and will dead lettered back to q1
• I published 3 messages that arrive in q1. 2 of them are very large (50 MB; probably will see the effect for smaller messages), so that I can see the segment file growing quickly.

Thus, the messages are endlessly looping from q1 to q2 and back to q1.

The segment file for q2 grows endlessly. With the rates described above, the file increases in size by about 250 MB every minute or so. After 25 or so minutes this is the segment file size on disk:

 ➜  rabbit2a@localhost ls -alh CHAD_RPZ013GOKC46E/*.segment
 -rw-r--r--  1 chadk  admin   5.3G Feb  7 16:49 CHAD_RPZ013GOKC46E/00000004.segment

I have tried specifically setting the dead letter routing key for each dead letter and leaving the routing key untouched. That does not appear to matter.

If there is a time when q2 has 0 messages, the segment file is deleted, and a new one starts. With the setup described above, I ensure that there is always at least 1 message in q2. There is no growth of segment file size in q1.

[michaelklishin]

So what is your question?

A segment file is not guaranteed to be deleted immediately when the queue reaches 0 messages in ready state.

Dead lettering has (and has always had) a loop detection mechanism but you can still construct a topology that would trick this basic protection mechanism.

I'm afraid I do not understand the meaning of

I have tried specifically setting the dead letter routing key for each dead letter and leaving the routing key untouched.

without an executable example.

Finally, messages of 50 MiB in size belong to a blob store. Then you can pass their ID(s) in the message.

[chadknutson]

Here is the question: why does the segment file grow endlessly?

I would expect that a new segment file would get created once it exceeds the default max size. That does not happen. I have seen one file grow to 100 GB.

Additionally, I would expect that once a message is dead lettered from q2 that it would no longer have a record of it. When that message comes back, it would be considered a 'new' message. Thus, there would be no growth of the segment files at all, save for the standard sawtooth growth and decay.

Regarding dead letter implementation question, I meant that I specifically set the dead letter routing key in 1 test, and I did not set it in another test. That made no difference.

[michaelklishin]

Dead lettering does keep track of message's history for several reasons:

• At least basic loop detection
• The x-death headers that it accumulates (that helps you reason about the history of the dead lettering events)

But none of the is particularly important since you implicitly assume that once the queue reaches 0 messages all of its on disk data is instantly gone. That's not the case, as my comment below tries to explain. Delivery acknowledgement is a critically important step in resource management of a quorum queue.

[michaelklishin]

You certainly can exhaust disk space with queue without consumers, TTL and length limit but with publishers (and DLX is one special case of a publisher).

The docs mention that QQ design assumes that eventually consumers do come online and begin consuming (and acknowledging deliveries).

The segment files are then compacted over time as consumers acknowledge deliveries. Compaction is
the process that reclaims disk space.

Quorum queues have certain protection against faulty consumers in place but they can only do so much,
and can only make so many assumptions about what the consumer intent is.

There are other protection mechanisms in place, most notably the delivery acknowledgement timeout, that protect from buggy consumers that consume and never acknowledge. #10500 is somewhat relevant.

If consumers in your system can go offline for long periods of time, then a queue length limit (they are very easy to reason about), or message TTL (a much more nuanced option), or both of them, are mandatory. Maybe even a TTL on the queue itself.

[chadknutson]

As I wrote, there are never consumers on q2. The number of messages on q2 is never large.

I get that quorum queues assume a consumer, but this design is not radical. It is simply using q2 as a delayed retry mechanism for messages that failed to be processed in q1.

You are suggesting that q2 should be a classic queue?

[michaelklishin]

I am suggesting that quorum queues have certain design decisions in place. Radical or not, a queue without consumers is not a case QQs, or any queue type, was designed for.

There are existing features that deal with such queues very well: queue length limits, message TTL, queue TTL.

Using a combination of DLX and TTL for delayed retries is good and all but this was never RabbitMQ's, or quorum queues' specifically, design goal.

[michaelklishin]

If you can mix different queue types, that would be one option. If not, then I'm afraid QQs alone may or may not be a good fit because a scenario where they never have any consumers, and DLX is the sole producer and "consumer", was never the design goal.

I've asked the rest of the team where during dead lettering we may be missing an acknowledgement (see below), because I see it performed in some dead lettering scenarios.

[michaelklishin]

Repeated requeues is not exactly the same scenario but there are similarities, and the QQ docs explicitly state that such consumers have a chance of growing the Raft log indefinitely, since they never acknowledge (usually not by design, of course).

You can use classic queues for these intermediary delays, they do not have a Raft-style log of operations and largely avoid this particular problem. They are also a non-replicated queue type since classic queue mirroring has been deprecated for years and this year, will be removed completely in RabbitMQ 4.x.

Whether using a classic queue is acceptable for such "message staging", I don't know, but these are the design space constraints.

If your system is entirely designed around such retries, you might be better off using a stream and re-consuming with the frequency that works best. At least technically they can be used as an alternative to retries.

Frameworks like NServiceBus likely support their timed retries magic by setting up a sequence of queues, and likely queues of the same type. That sucks but if a queue never has any consumers and acknowledgements but has a message flow, it will run into the issue described in the doc section above.

I see acknowledgements performed in the DLX worker module. If you can put together an executable way to reproduce, we should be able to trace where it is or is not called.

[chadknutson]

I am happy to put something together for testing. This is a pretty simple test to set up. But I have not put together a docker or whatever executable before, so it might take me a day or so.

[michaelklishin]

No rush. We only really need a Python script (or a Java app, or whatever you prefer and we have enough expertise with to run) that would demonstrate the topology and how you publish the messages. Unless you modify Raft-based settings in rabbitmq.conf, in that case we'd need the entire config file, too.

[chadknutson]

I have attached python scripts for publisher and consumer. This is a slightly different implementation than described above, but I see the same lack of clearing segment files in both cases.

Here a publisher is constantly publishing messages to a 'target' queue. A consumer is subscribed to the 'target' queue, but rejects 1 out of 50 messages. The rejected msgs are dead lettered to the 'delay' queue, which has a TTL of 10 seconds and dead letters the messages back to 'target'. Segment files grow in the 'delay' queue folder
pub_retry_def.txt
cons_retry.txt

[chadknutson]

And just for fun, here is another case that shows the same behavior: using a quorum queue as a delay mechanism. I publish to the 'delay' queue, which has a TTL of 10 seconds; messages are dead lettered to the 'target' queue that has a consumer.

Note that I am only using very large messages in my scripts so that I can see the effect more quickly. I imagine that small messages would have the same effect, but it would take longer to see (the wal file would a long time to fill up!).
cons.txt
pub.txt

[kjnilsson]

You need to set a much lower than the default raft.max_segment_entries configuration. This controls how many entries will be added to each segment. The default is 4096. I suggest you try 128.

[chadknutson]

I set the value of raft.max_segment_entries to 96 to speed up results. What happens here is that segment files are rotated, but older files are not removed.

With a little remote_shell evaluation, we can see that files have only 96 segments, but the number is increasing over time

(rabbit2a@localhost)11> [{File, To - From + 1} || {From, To, File} <- (maps:get(log,element(3, element(2, recon:get_state(element(1, amqqueue:get_pid(Q)))))))#ra_log.reader
#ra_log_reader.segment_refs].                                                                                                                                               
[{"00000029.segment",83},                                                                                                                                                   
 {"00000028.segment",96},                                                                                                                                                   
 {"00000027.segment",96},                                                                                                                                                   
 {"00000026.segment",96},                                                                                                                                                   
 {"00000025.segment",96},                                                                                                                                                   
 {"00000024.segment",96},                                                                
 {"00000023.segment",96},                                                                
 {"00000022.segment",96},                                                                
 {"00000021.segment",96},                                                                
 {"00000020.segment",96},                                                                
 {"00000019.segment",96},                                                                
 {"00000018.segment",96},                                                                
 {"00000017.segment",96},                                                                
 {"00000016.segment",96},                                                                
 {"00000015.segment",96},  
 {[...],...},                              
 {...}] 

[chadknutson]

Oh, I should have waited a bit longer!
I see that segment files are deleted once there are either 42 or 43 of them. That is good news!

[chadknutson]

For completeness, I ran some tests for different values of segment_max_entries. The number of segment files kept on disk decreases with increasing allowed entries.

segment_max_entries.       number of segments before deletion
50                            82
96                            43
200                           20
2000                          2

This remains a bit curious, since the total amount of disk usage appears to be invariant. Or maybe there is another parameter that could be altered, such as wal_max_size_bytes? I have been using the default value (536870912)