3.13: Consumer runs into an exception when the x-deaths header is populated

[jeremie-seguin]

Describe the bug

After upgrading RMQ server to version 3.13.7 from version 3.12.14 and enabling the following feature flags:

quorum_queue_non_voters 
message_containers 
detailed_queues_endpoint
stream_filtering
stream_update_config_command
message_containers_deaths_v2

Consuming message results in the following exception:

** Generic server <0.59804052.0> terminating
** Last message in was {'$gen_cast',
                           {method,
                               {'basic.get',0,<<"retry.queue">>,false},
                               none,noflow}}
** When Server state == {ch,
                         {conf,running,rabbit_framing_amqp_0_9_1,1,
                          <0.59804017.0>,<0.59804017.0>,<0.59803933.0>,
                          <<"<rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq.1728269515.59803933.0>">>,
                          undefined,
                          {user,<<"admin">>,
                           [administrator],
                           [{rabbit_auth_backend_internal,
                             #Fun<rabbit_auth_backend_internal.3.26105038>}]},
                          <<"main">>,<<>>,<0.59804002.0>,
                          [{<<"publisher_confirms">>,bool,true},
                           {<<"exchange_exchange_bindings">>,bool,true},
                           {<<"basic.nack">>,bool,true},
                           {<<"consumer_cancel_notify">>,bool,true},
                           {<<"connection.blocked">>,bool,true},
                           {<<"authentication_failure_close">>,bool,true}],
                          none,0,134217728,1800000,#{},infinity,1000000000,
                          false},
                         {lstate,<0.59804041.0>,false},
                         none,1,
                         {0,[],[]},
                         {state,#{},erlang},
                         #{},#{},
                         {state,fine,5000,#Ref<0.1133642180.4178575361.62869>},
                         false,1,
                         {rabbit_confirms,undefined,#{}},
                         [],[],none,flow,[],
                         {rabbit_queue_type,#{}},
                         #Ref<0.1133642180.4178575361.62864>,false}
** Reason for termination == 
** {{bad_generator,<<>>},
    [{mc_amqpl,'-death_table/7-lc$^0/1-0-',1,
               [{file,"mc_amqpl.erl"},{line,814}]},
     {mc_amqpl,death_table,7,[{file,"mc_amqpl.erl"},{line,814}]},
     {lists,map,2,[{file,"lists.erl"},{line,1559}]},
     {mc_amqpl,deaths_to_headers,2,[{file,"mc_amqpl.erl"},{line,770}]},
     {mc_amqpl,protocol_state,2,[{file,"mc_amqpl.erl"},{line,407}]},
     {rabbit_channel,handle_basic_get,7,
                     [{file,"rabbit_channel.erl"},{line,2723}]},
     {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,659}]},
     {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]}]}

This exception is occuring on most queues but not on all messages, I can't visualize offending messages as the issue is happening with every clients and the managment interface / shovel. It did not happend after the upgrade itself but only when the feature flags were enabled.

Our queues are mostly quorum queues and classic mirrored queues for dead-letter.
We are running rabbitmq using the cluster-operator v2.0.0 on 3 nodes

Reproduction steps

I was able to reproduce the issue on two separate environment with an important backlog of messages. I did not reproduce it on a test environment.

1- Upgrade from 3.12.14 to 3.13.7
2- enable new feature flags
3- consume messages

Expected behavior

No exception

Additional context

Running a fresh 3.13 cluster with the same configuration / queues do not produce the error, only cluster upgraded from 3.12 are having this issue

[michaelklishin]

@jeremie-seguin we need an executable way to reproduce. There are many thousands of clusters that do not exhibit this behavior (3.13.x has been around for about six months), and we cannot guess what is different about your messages, topology, and so on. We do not guess in this community.

FYI, RabbitMQ 3.13. is out of community support so we also need a way to reproduce against 4.0.2.

[michaelklishin]

Plus there is a number of PRs already finished and merged for 4.0.3.

[michaelklishin]

@jeremie-seguin finally, we will need all feature flags except for khepri_db to be enabled on 4.0.x. Except for Khepri, we expect the users to enable all feature flags, not just a handful of them selectively. As the docs say, they are not optional features, they are an upgrade safety mechanism, so some of them address "bugs" or behavior that is not a feature.

[ansd]

I couldn't reproduce this issue. Please, be more specific in your reproduction steps or provide executable scripts.
Missing infos include:

• Which queue types are used in source queue and dead letter queue?
• Which dead letter exchange is used?
• Which dead letter routing key is used?
• Which dead letter strategy is used? at-most-once or at-least-once?
• How are dead letter queues bound to the dead letter exchange?
• Which protocol was the message published with?
• ...

[jeremie-seguin]

Hello I'm trying to reproduce the error on a simpler configuration that I can export.

Our source queue are quorum or classic depending on the usage, all queue use the same exchange dead.letter.exchange and the same routing key retry . A dedicated consumer will then redirect the message on a queue with a ttl and the original queue as a dead letter so the message can be retried after a delay or quarantined. All our client use AMQP 0.9.1.

[kjnilsson]

@jeremie-seguin do any of your applications modify or append to the x-death header?

[jeremie-seguin]

We indeed put an ArrayList in the MessageProperties then send it using spring-amqp

In the header we set a value for reason, exchange, queue, time and count but routing-keys is defined but empty

[ansd]

What exactly means "routing-keys is defined but empty"?
The routing-keys field must be an AMQP 0.9.1 array of longstr.
I documented this in the table in https://www.rabbitmq.com/docs/dlx#effects
Note that at least one routing key must be set in this array (even if it's the empty string <<"">>).

[jeremie-seguin]

Here is am example of the message sent when we modify the header

import pika

if __name__ == '__main__':
    connection = pika.BlockingConnection(pika.ConnectionParameters('localhost', virtual_host='/'))
    channel = connection.channel()
    channel.queue_declare(queue='hello', durable=True)
    headers = {
        'x-death': [{
            'reason': 'rejected',
            'routing-key': '',
            'exchange': 'example.exchange',
            'queue': 'example.queue',
            'time': '2023-10-09T12:00:00Z',
            'count': 5,
        }]
    }

    channel.basic_publish(exchange='',
                          routing_key='hello',
                          properties=pika.BasicProperties(
                              delivery_mode=pika.spec.PERSISTENT_DELIVERY_MODE,
                              headers=headers
                          ),
                          body='Hello')
    connection.close()

I sent this message on a 3.13.0 instance using this header, then consumed it, the message contained the header as sent.
I did the same on a 3.13.14 instance and the message contained an empty header.

I could not keep the queue data produced on the 3.13.0 after upgrading to the 3.13.14 but it looks like the issue is indeed comming from our usage of this header. We will rework how we use the dead letter to not have to override the header.

Thank you very much for your insights !

[ansd]

Thanks for the sample code.
Yes, the following part looks incorrect:

'routing-key': '',

[ansd]

Also going forward, it's best to avoid having clients to (re-)publish with x-death header set as this x-header won't be interpreted anymore starting with RabbitMQ 4.0:
https://github.com/rabbitmq/rabbitmq-server/blob/main/release-notes/4.0.1.md#breaking-changes-and-compatibility-notes