x-death header count is no longer incremented from RabbitMQ version 3.13.0+

[cressie176]

Describe the bug

Prior to v13.13.0, when a message is rejected and routed to a dead letter queue, the broker maintains an "x-death" header added similar to the following...

[
  {
    "count": 1,
    "reason": "rejected",
    "queue": "q1",
    "time": {
      "!": "timestamp",
      "value": 1716827089
    },
    "exchange": "",
    "routing-keys": [
      "q1"
    ]
  }
]

if the message is moved from the dead letter queue back the work queue, and again rejected, the count property is updated by the broker, i.e.

[
  {
    "count": 2,
    "reason": "rejected",
    "queue": "q1",
    "time": {
      "!": "timestamp",
      "value": 1716827089
    },
    "exchange": "",
    "routing-keys": [
      "q1"
    ]
  }
]

From RabbitMQ v13.13.0 the count is no longer updated. I'm author of a RabbitMQ client for Node.js called Rascal, and was about to rely on the incrementing count for this feature.

There isn't any mention of this breaking change in the v13.13.0 release notes, but I see there is a related issue. Similar to the author of this issue, I am also not editing the x-death header, but am relying on the broker maintaining the count. I am unable to maintain this in a custom header since I would need to modify the message on rejection, which is not possible. If the decision is for x-death to be "broker only" and to ignore this header when published from the client, that's no problem, but to maintain backwards compatibility, the broker must count the rejections.

In #10709 you also advise that RabbitMQ v4 is likely to make header's starting with 'x-' broker only. Thankfully none of Rascal's custom headers take this format, but wouldn't this mean that any messages with custom 'x-' headers that are in flight / enqueued when the broker is upgraded might be broken?

Reproduction steps

1. Configure a work queue with a dead letter exchange / dead letter queue
2. Publish a message to the work queue
3. Reject the message with requeue false
4. Move the message to the work queue
5. Rerject the message with requeue false again

The message should have an 'x-death' record with a count of 2, but instead it has a count of 1

Expected behavior

The x-death record's count property should be incremented whenever a message is rejected.

Additional context

No response

[michaelklishin]

There is a set of changes related to dead lettering that have shipped in 3.13.2 or will ship in 3.13.3: #10709,
#11173, #11160, #11159, #11174

[michaelklishin]

As for x-* headers, nothing has been decided yet. There's a set of headers such as x-deaths that were never meant to be modified by applications.

[cressie176]

@michaelklishin thank you for the link to #11174, it provides a lot more context, however I think it is still a bug and should not have been closed as a duplicate.

There's a set of headers such as x-deaths that were never meant to be modified by applications.

The application is not modifying these, other than when they move the message (by copying it) to a different queue.

Irrespective it's very confusing behaviour to have a "count" property set by the broker, which until recently was incremented with each rejection, but now if fixed at 1 no matter how many times the message is rejected. This was an undocumented change in v3.13.0, that breaks the current documented behaviour.

count: how many times this message was dead-lettered in this queue for this reason

[michaelklishin]

@cressie176 since 3.13.0 there were four changes directly related to dead lettering, including at least one related to the conditions under which the counter is incremented. They are all mentioned in release notes (including those pending release under release-notes in this repository).

Use https://hub.docker.com/layers/pivotalrabbitmq/rabbitmq/v3.13.x/images/sha256-b32c328236bec5ec2b94bd4afd290687844a957919fe838457d10858c620bfc4?context=explore in order to compare.

If this is something distinct from all those behaviours, please put together an executable way to reproduce and we will consider reopening this issue or filing a new one.

[cressie176]

I searched the v3.13.0 release notes (the version which introduced the bug) prior to reporting it. They only mentioned dead lettering once and give no indication there was a breaking change relating to the way rejections are counted...

Messages are now internally stored using a new common heavily AMQP 1.0-influenced container format. This is a major step towards a protocol-agnostic core: a common format that encapsulates a sum of data types used by the protocols RabbitMQ supports, plus annotations for routing, dead-lettering state, and other purposes.

[cressie176]

@michaelklishin

since 3.13.0 there were four changes directly related to dead lettering, including at least one related to the conditions under which the counter is incremented

#10709 - This is the same bug, although it manifests with message expiry. The OP was able to workaround, but noted how this would likely cause issues for other users. @lukebakken acknowledges it is a regression and breaking change, but the issue was closed regardless.

#11173 - This issue is about adding the x-death header to streamed messages

#11160 - This issue is about a warning message when dead letter messages are dropped by the broker

#11159 - This issue is about dead letter messages being incorrectly dropped by the broker

#11174 - This discusses approaches for avoiding infinite dead letter cycles, and the introduction of deaths_v2 annotation

None of the above, other than the first which was closed without resolving the bug, are concerned with v3.13.0 of the rabbitmq broker failing to increment the x-death count field.

If this is something distinct from all those behaviours, please put together an executable way to reproduce and we will consider reopening this issue or filing a new one.

https://github.com/cressie176/rabbitmq-11331 demonstrates the bug

[michaelklishin]

@cressie176 apparently it was an intentionally decision (in the same discussion as #10709 (comment)) by several team members to keep the current behavior instead of reverting it only to change it back for 4.x.

Thank you for taking the time to put together an executable example to reproduce!