Shovel with an AMQP 0-9-1 source and an AMQP 1.0 destination runs into an exception

[adenoeud]

Hello,

I recently upgrade my Rabbitmq cluster from 3.12.9 to 3.13.1.

When I push a message to the queue linked to one of my shovel through a java client, I have this error :

shovel_issue.txt

The same message on the Rabbitmq version 3.12.9 is well sent through the shovel.

However, when I push a message manually on the rabbitmq admin webapp everything is ok on the 2 versions.

I think, it's maybe because of the source/destination protocol configured on my shovel (0.9.1 for the source / 1.0 for the destination) but I'm not sure about it.

Some links on a potential change on the protocol conversions that is new on the release 3.13.0 :

https://www.rabbitmq.com/docs/conversions#amqpl-amqp
https://www.rabbitmq.com/blog/2024/03/11/rabbitmq-3.13.0-announcement#message-containers

If someone have an idea on it, I will be happy to discuss about it !

[michaelklishin]

@adenoeud we need a specific set of steps or an executable way to reproduce this against 3.13.4. A short log message tells nothing about the content of the message, for example, or any of the ≈ 20 Shovel settings.

[adenoeud]

Hello @michaelklishin,
Thanks for your answer !

Here are my config :

Rabbitmqadmin version

rabbitmqadmin 3.8.9

Shovel

rabbitmqadmin -V / declare parameter -u $RABBITMQ_USERNAME -p $RABBITMQ_PASSWORD -P $SERVER_MANAGEMENT_PORT -N $SERVER_NODE_NAME component=shovel name=outbound_msg value='{
"src-protocol" : "amqp091",
"src-uri": "amqp://",
"src-queue": "source-queue",
"dest-protocol" : "amqp10",
"dest-uri" : "amqps://'"$DESTINATION_AZURE_LOGIN"':'"$DESTINATION_AZURE_PASSWORD"'@'"$DESTINATION_AZURE_SERVER_TO_CONNECT"'",
"dest-address": "'"$DESTINATION_AZURE_DEST_ADDRESS"'",
"ack-mode": "on-confirm",
"src-delete-after": "never"
}'

Binding

/usr/local/bin/rabbitmqadmin -N $SERVER_NODE_NAME -P $SERVER_MANAGEMENT_PORT -u $RABBITMQ_USERNAME -p $RABBITMQ_PASSWORD declare binding source=source-exchange destination_type=queue destination=source-queue

Exchange

/usr/local/bin/rabbitmqadmin -N $SERVER_NODE_NAME -P $SERVER_MANAGEMENT_PORT -u $RABBITMQ_USERNAME -p $RABBITMQ_PASSWORD declare exchange name=source-exchange type=direct arguments={"alternate-exchange":"ha.alternate-exchange"}

Queue

/usr/local/bin/rabbitmqadmin -N $SERVER_NODE_NAME -P $SERVER_MANAGEMENT_PORT -u $RABBITMQ_USERNAME -p $RABBITMQ_PASSWORD declare queue name=source-queue durable=true

I pushed a small spring application to send a message to the queue to reproduce the issue
https://github.com/adenoeud/testrabbitmq

Tell me if you need more details

[adenoeud]

Hello @michaelklishin,
I found the issue, the message property priority is set by default by Spring to 0.
I removed it from the message properties and it worked !

(I found in the doc that the priority have to be set from 1 to 255, and it works aswell by setting a good value).

But I have the error only when I send a message through a shovel.

[michaelklishin]

@adenoeud thanks. Can you share a bit more about how you use Spring AMQP and what did you have to do to override the priority? I can chat to the Spring AMQP maintainer about whether this default is optimal, besides looking at what may be tripping up Shovel in this case.

[adenoeud]

I pushed something here with the way I fixed the issue : https://github.com/adenoeud/testrabbitmq

It's a simple Spring AMQP client that send a message to a queue

[michaelklishin]

@adenoeud @kjnilsson has found what is likely the root cause: our Erlang AMQP 1.0 client sets the wrong type for the priority field, and Azure rejects such messages.

This was addressed for 3.12.x but not 3.13.x, apparently. It will be soon enough (#11728).