annotate_msg seems to be calling set_annotation with wrong arguments

[gabrielrat]

Describe the bug

This line
https://github.com/rabbitmq/rabbitmq-server/blame/788879969ed10120b61bf03e6d2f72b1447ca73b/deps/rabbit/src/rabbit_fifo.erl#L1837
mc:set_annotation(delivery_count, DelCount, Msg);
seems to be calling
mc:set_annotation which doesnt have any clause for delivery_count atom. I am new to Erlang and I might be wrong but my first guess would be that this should be the fix
mc:set_annotation(<<"x-delivery-count">>, DelCount, Msg);

Noticed this bug because no consumer can consume from a quorum queue that contains high delivery count and this error occurs

2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0> ** {function_clause,
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>        [{mc_compat,set_annotation,
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>             [delivery_count,1826,
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>              {basic_message,
XXXXXXXXXXXXXXXXXXXXXX
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>                  <<>>,false}],
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>             [{file,"mc_compat.erl"},{line,61}]},
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>         {rabbit_fifo_client,add_delivery_count_header,2,
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>             [{file,"rabbit_fifo_client.erl"},{line,228}]},
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>         {rabbit_fifo_client,'-transform_msgs/3-fun-0-',3,
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>             [{file,"rabbit_fifo_client.erl"},{line,843}]},
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>         {lists,map,2,[{file,"lists.erl"},{line,1559}]},
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>         {rabbit_fifo_client,handle_delivery,4,
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>             [{file,"rabbit_fifo_client.erl"},{line,777}]},
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>         {rabbit_queue_type,handle_event,3,
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>             [{file,"rabbit_queue_type.erl"},{line,588}]},
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>         {rabbit_channel,handle_cast,2,
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>             [{file,"rabbit_channel.erl"},{line,699}]},
2024-09-26 15:33:43.440384+00:00 [error] <0.111012.0>         {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]}]}

Reproduction steps

1.Setup a quorum queue which has messages with high delivery count on RabbitMQ 4.0.2 server.
2.Try to consume from that queue with a client or a shovel.

Expected behavior

The message should be consumed, but it just causes an 541 error.

Additional context

No response

[michaelklishin]

@gabrielrat can you please share an executable way of reproduce this exception?

[ansd]

I'm able to reproduce this error as follows.

1. Start RabbitMQ v3.13.7 with feature flag message_containers disabled:

make run-broker TEST_TMPDIR="$HOME/scratch/rabbit/test" RABBITMQ_FEATURE_FLAGS=quorum_queue,implicit_default_bindings,virtual_host_metadata,maintenance_mode_status,user_limits,feature_flags_v2,stream_queue,classic_queue_type_delivery_support,classic_mirrored_queue_version,stream_single_active_consumer,direct_exchange_routing_v2,listener_records_in_ets,tracking_records_in_ets

In the Management UI
2. Create a quorum queue with x-delivery-limit=10
3. Publish a message to this queue.
4. Requeue this message two times.
5. ./sbin/rabbitmqctl enable_feature_flag all
6. Stop the node
7. git checkout v4.0.2
8. make run-broker TEST_TMPDIR="$HOME/scratch/rabbit/test"
9. Again in the Management UI, Get Message with Automatic Ack leads to above crash:

2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0> ** Reason for termination ==
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0> ** {function_clause,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>        [{mc_compat,set_annotation,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>             [delivery_count,2,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>              {basic_message,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                  {resource,<<"/">>,exchange,<<>>},
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                  [<<"qq1">>],
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                  {content,60,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                      {'P_basic',undefined,undefined,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                          [{<<"x-delivery-count">>,long,2}],
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                          2,undefined,undefined,undefined,undefined,undefined,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                          undefined,undefined,undefined,undefined,undefined},
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                      none,none,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                      [<<"m1">>]},
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                  <<230,146,94,58,177,125,64,163,30,18,177,132,53,207,69,103>>,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>                  true}],
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>             [{file,"mc_compat.erl"},{line,61}]},
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>         {rabbit_fifo_client,add_delivery_count_header,2,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>             [{file,"rabbit_fifo_client.erl"},{line,228}]},
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>         {rabbit_fifo_client,dequeue,4,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>             [{file,"rabbit_fifo_client.erl"},{line,211}]},
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>         {rabbit_queue_type,dequeue,5,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>             [{file,"rabbit_queue_type.erl"},{line,755}]},
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>         {rabbit_misc,with_exit_handler,2,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>             [{file,"rabbit_misc.erl"},{line,526}]},
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>         {rabbit_channel,handle_method,3,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>             [{file,"rabbit_channel.erl"},{line,1257}]},
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>         {rabbit_channel,handle_cast,2,
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>             [{file,"rabbit_channel.erl"},{line,629}]},
2024-09-30 09:22:59.545676+02:00 [error] <0.1185.0>         {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]}]}

[gabrielrat]

Unfortunately, I had to delete the queue with the messages that were causing the errors and was not able to reproduce it again but I am glad @ansd was able to do it. I was suspecting that it has something to do with the Rabbit version upgrade because I did one before this happened.

[ansd]

@gabrielrat thank you for reporting this bug.
The fix will be available in RabbitMQ 4.0.3.

[gabrielrat]

@ansd Thank you as well for the quick fix! 🙏