Quorum queue message order issue when consumers disconnect

[anhanhnguyen]

Hi,

We found an unexpected behaviour with quorum queues when doing get messages, nack and requeue.

The testing procedure:

1. Consume messages from a quorum queue with a prefect count = 5. Do not acknowledge the messages
2. Kill the consumer (drop the connection)
3. Get messages from UI with nack and requeue true

When we get the messages from the UI, we expect to get the first message in the queue and it should be the same message every time. However, we get different messages each time.

Example:
Publish messages 1, 2, 3, 4, 5, 6, 7, 8.
Consume with prefetch = 5, do not acknowledge.
Kill the consumer.

Get a message from the queue.
Returns message 1.
Get a message from the queue.
Returns message 2.
Get a message from the queue.
Returns message 3.
Etc until message 5, then it starts from 1.

Expectation would be to get the message 1 for each basic-get as long as it's not acknowledged. This happens even if delivery-limit is set.

We are preparing a test to show this behaviour as well.

Thank you,
Anh

[michaelklishin]

Quorum queues starting with 3.10 have two ways of requeueing by design: to avoid unbound disk log growth with pathological consumers that consume and never acknowledge.

basic.get via the UI is not how you should be assessing message ordering. With concurrent consumers and requeueing, message ordering becomes a moot point: the queue will try to put it back into the original location but this can be immediately negated by operations of/on other consumers. They avoid this by requeueing back to the end of the queue instead of the original position (e.g. the head).

[kjnilsson]

Yes correct the only way to really ensure ordering is to configure the queue to use single active consumer and ensure there is a delivery-limit set either via a declare arg or a policy.

[kjnilsson]

See https://www.rabbitmq.com/quorum-queues.html#repeated-requeues

[anhanhnguyen]

Hi,

Yes, we did the test with delivery-limit set and single active consumer.

I think this behaviour is from a list keeping track of requeued messages returns

When getting messages, messages from the head of returns will be taken, and the message is placed back on "Nack" (or requeued when consumers are down) to the end of this list.

By this way, the messages will be different each basic.get and nack-requeue true (messages cycle in the returns list)

We made a simple test here:
main...anhanhnguyen:rabbitmq-server:main

We would like to confirm whether this is the intended design as we still expect the messages to go back to the "original order".

Thank you and best regards,
Anh

[kjnilsson]

You mean if the single reader returns messages in an order different to the one they were received they would be ordered in the order the were returned?

[rjnilsson]

That was pretty hard to parse :)

I don't know about that use case. I forgot to mention that we don't actually requeue explicitly; when we detect logic errors we simply bail out, the service dies, and the container is restarted. So just dropping the connection and expecting the original order again when reconnecting.

[kjnilsson]

For a single consumer that crashes any pending messages will be returned in the order they were assigned to the consumer. Are you seeing something different here?

[kjnilsson]

So we have a queue with [A, B, C, D, E] ready messages. The queue has a delivery-limit set and is configured to use single active consumer. A consumer attaches with a prefetch of 3 and gets assigned [A, B, C]. The consumer acks A then crashes. B and C gets added to the internal return queue in the order they were received, i.e. [B, C]. Another consumer attaches and gets assigned [B, C, D] as expected. That is how it should work. Do you see something different in your tests?

[rjnilsson]

Yes, our observation is different. I'll try with a more specific example:

• Quorum queue with delivery-limit (and max-length) with three nodes (IIRC)
• Single consumer
• Prefetch of 3 (this is not the actual prefetch value that we've been using)

Initial scenario:

1. Consumer connects to queue
2. Messages arrive [A, B, C, D, E]
3. Consumer receives [A, B, C] into prefetch (of course, this happens concurrently with 2 above)
4. Consumer processes and acks A, but crashes when attempting to process B
5. Consumer starts up, connects to queue
6. Consumer receives [B, C, D] into prefetch (actual order of reception not verified, but we order them internally)
7. Consumer crashes when attempting to process B
8. Return to 5 above

The above continues for some undefined time. We then turn off consumer auto-restart, and:

1. Start the web admin interface
2. Open the queue
3. Get single message from the queue (nack requeue true) and seemingly receives one random message of the set [B, C, D]
4. Get single message once again (nack requeue true) and seemingly receives some other random message of the set [B, C, D]
5. Etc

The "randomness" above are only illustrations (sometimes we actually see B first, but then something else), but I fail to see any pattern. We would expect to consistently receive B each time, as was our observation when using classic mirrored queues.

I also think that the example from @anhanhnguyen above shows this in a similar way.

Hope this helps.

[kjnilsson]

In a quorum queue issuing a basic.get is effectively implemented as a consume with a prefetch of one. This is why basic.get is disallowed when using single active consumer.

…

On Sun, 22 Jan 2023 at 17:26, rjnilsson ***@***.***> wrote: Yes, our observation is different. I'll try with a more specific example: - Quorum queue with delivery-limit (and max-length) with three nodes (IIRC) - Single consumer - Prefetch of 3 (this is not the actual prefetch value that we've been using) Initial scenario: 1. Consumer connects to queue 2. Messages arrive [A, B, C, D, E] 3. Consumer receives [A, B, C] into prefetch (of course, this happens concurrently with 2 above) 4. Consumer processes and acks A, but crashes when attempting to process B 5. Consumer starts up, connects to queue 6. Consumer receives [B, C, D] into prefetch (actual order of reception not verified, but we order them internally) 7. Consumer crashes when attempting to process B 8. Return to 5 above The above continues for some undefined time. We then turn off consumer auto-restart, and: 1. Start the web admin interface 2. Open the queue 3. Get single message from the queue (nack requeue true) and seemingly receives one random message of the set [B, C, D] 4. Get single message once again (nack requeue true) and seemingly receives some other random message of the set [B, C, D] 5. Etc The "randomness" above are only illustrations (sometimes we actually see B first, but then something else), but I fail to see any pattern. We would expect to consistently receive B each time, as was our observation when using classic mirrored queues. I also think that the example from @anhanhnguyen <https://github.com/anhanhnguyen> above shows this in a similar way. Hope this helps. — Reply to this email directly, view it on GitHub <#6805 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAHFBHDLWWLWXBS6TDIHLWTVUTJANCNFSM6AAAAAATR3YRXM> . You are receiving this because you commented.Message ID: ***@***.*** com>

-- *Karl Nilsson*

[rjnilsson]

I think we are still in a bit of a misunderstanding. Quorum queues works perfectly fine during normal usage for us. The ordering semantics are ok with respect to that.

This is related to attempting to peek at (and later remove) the head message from the queue using the admin interface without affecting the order of the currently existing messages, including the head message. This at a time when no subscribers except the admin interface interacts with, or are even attached to, the queue.

Peeking at the first message to see the contents ("nack + requeue true"), and if the contents is "bad" then remove it using "reject + requeue false", seems to be impossible with quorum queues. Kind of like the observer effect in physics, at least when judging by the externally observable behavior and disregarding implementation details.

We did not notice this behavior before switching to quorum queues. So, maybe my point at this time is simply that the behavior of getting a single messages from the queue, using the admin interface, when messages have been redelivered, seem to have changed. And that I can't find any documentation on that.

[mkuratczyk]

I've tried multiple times but was unable to reproduce. Perhaps I'm missing something here, this is what I tried:

# declare the queue
rabbitmqadmin declare queue name=qq queue_type=quorum arguments='{"x-delivery-limit": 5}'

# publish messages with payload from 'A' to 'Z'
for PAYLOAD in {A..Z}; do rabbitmqadmin publish routing_key=qq payload=$PAYLOAD; done

# consume with a prefetch of 3, wait 10 seconds before acking - this gives us time to kill the process
perf-test -x 0 -qq -u qq -qa x-delivery-limit=5 -q 3 -L 10000000

# kill the publisher after the first message is acked (this part is manual and time sensitive, of course could be replaced with a small app)
killall -9 java

# at this point I have 25 messages in the queue ('A' was acked, 'B' and 'C' requeued)
rabbitmqadmin get queue=qq ackmode=reject_requeue_true
+-------------+----------+---------------+---------+---------------+------------------+-------------+
| routing_key | exchange | message_count | payload | payload_bytes | payload_encoding | redelivered |
+-------------+----------+---------------+---------+---------------+------------------+-------------+
| qq          |          | 24            | B       | 1             | string           | True        |
+-------------+----------+---------------+---------+---------------+------------------+-------------+

(note, while the output says there are 24 messages in the queue, since the message is requeued, there are actually 25 again)

I also tried with the management UI but I always got 'B' as well (I believe they perform the same action under the hood).
If you can reproduce your issue using these steps or modify them accordingly, or write an app that demonstrates the problem, it'd be much easier to help/fix the issue.

[rjnilsson]

Thanks for helping out. I do have an active example of the behavior "in the wild" right now, that exposes the "different messages every get through the admin interface" but that is internally.

Did you see the sample reference earlier by @anhanhnguyen at main...anhanhnguyen:rabbitmq-server:main

[mkuratczyk]

Thanks, I missed that. I'll leave it to Karl to decide whether this is a bug or a feature.

[rjnilsson]

Sounds great. Thanks to you both.