Quorum dead-letter queues with at-least-once delivery cannot deliver previously dead-lettered items without routing key

[Lukazoid]

Describe the bug

When using no routing keys and transitioning from an at-most-once to an at-least-once dead-letter strategy with quorum queues a situation was observed where previously dead-letter items could get stuck in the internal dead-lettered bucket despite a valid route existing.

Reproduction steps

1. Create quorum queue foo_deadletter
2. Create quorum queue foo with argument x-dead-letter-exchange: foo_deadletter
3. Create direct exchanges foo and foo_deadletter
4. Create binding from foo -> foo with no routing key
5. Create binding from foo_deadletter -> foo_deadletter with no routing key
6. Publish items to foo with no routing key
7. Consume and reject all items from foo
8. Confirm they end up in foo_deadletter
9. Create policy:

{
    "vhost": "/",
    "name": "at-least-once-deadletter",
    "pattern": "foo$",
    "apply-to": "quorum_queues",
    "definition": {
        "dead-letter-strategy": "at-least-once",
        "overflow": "reject-publish"
    },
    "priority": 1
}

10. Confirm policy applied to foo
11. Shovel items from foo_deadletter -> foo
12. Consume and reject all items from foo
13. See that items are stuck in foos internal dead-lettered bucket and cannot be routed
14. Observe the log message Cannot forward any dead-letter messages from source quorum queue 'foo' in vhost '/' with configured dead-letter-exchange exchange 'foo_deadletter' in vhost '/' and configured dead-letter-routing-key 'undefined' in rabbitmq server logs

Steps to resolve:

1. Create binding from foo_deadletter -> foo_deadletter with a routing key of dlx
2. Update policy (note the dead-letter-routing-key):

{
    "vhost": "/",
    "name": "at-least-once-deadletter",
    "pattern": "foo$",
    "apply-to": "quorum_queues",
    "definition": {
        "dead-letter-strategy": "at-least-once",
        "overflow": "reject-publish", 
        "dead-letter-routing-key": "dlx"
    },
    "priority": 1
}

3. Observe that items eventually get routed

Expected behavior

The items should have been routed after step 12 above as they should not have any routing key and there is no dead-letter-routing-key and the foo_deadletter -> foo_deadletter binding does not want a routing key.

Additional context

Using RabbitMQ 3.13.1

[Lukazoid]

The code lines which lead to a workaround was in rabbit_fifo_dlx_worker so perhaps the issue is the use of OriginalRoutingKeys when there are none?

[michaelklishin]

@Lukazoid there were multiple dead-lettering-related changes since 3.13.1. Multiple in 3.13.3 alone, in fact.

[michaelklishin]

Create binding from foo_deadletter -> foo_deadletter with a routing key of dlx

I'm afraid I do not understand what is being bound to what. Can we please use names more sensible that "foo", and not name every single thing "foo" (an exchange, a policy, etc)? Even non-repeated single letter names are more descriptive.

"Create this foo then that foo than bind it to this other foo which has a policy named foo" is not the kind of reproduction steps anyone wants to use.

If the steps are specific, you are welcome to put together an executable test case using any client library you like.

[michaelklishin]

@ansd is aware of this discussion but I'll mention him just in case.

[ansd]

11. Shovel items from foo_deadletter -> foo

Shovelling this message will change the message's exchange to the default exchange "" and the routing key to foo.
Therefore, the following expected behaviour description is wrong:

The items should have been routed after step 12 above as they should not have any routing key

Hence, there is no bug and at-least-once dead lettering works as expected.
In fact, this situation shows the strength of at-least-once dead lettering: Despite the misunderstanding and the resulting misconfiguration of the topology where sending to the dead letter exchange with no appropriate route would have resulted in the message being dropped with the default at-most-once dead lettering strategy, at-least-once dead lettering keeps the message safe in the source quorum queue until the user (in this case you in your described "Steps to resolve") fixes the topology.

[Lukazoid]

Ah, shovelling changes the routing-key @ansd ? Maybe that's the part I was missing

@michaelklishin it was meant to just be a trivial case to represent a 1-1 exchange-queue binding, sorry if it confused matters.

[michaelklishin]

Shovel consumes from its source and re-publishes to the destination, using the default exchange by default. You can set the routing key as a constant but shovels won't try to preserve the routing key.

Apparently the docs claim that they preserve the routing key but that's not been the case since the major Shovel refactoring to support AMQP 1.0 a few years ago. I will update the docs.

[michaelklishin]

This is even more involved. What destination and "method" Shovels use during republishing depends on different combinations of settings: dest-exchange vs. dest-queue, for example. But the idea remains the same: shovels may or may not preserve the routing key of the original message.

[ansd]

Ah, shovelling changes the routing-key @ansd ? Maybe that's the part I was missing

Yes