Shovel tries to re-publish a message that was refused by target

[fgheysels]

We have rabbitmq with the shovel plugin installed on a number of vessels.

The shovel is configured to transfer messages from a rabbitmq queue to a queue in Azure ServiceBus. We have seen a few cases where a message that was submitted on the rabbitmq queue had a payload > 256KB. Azure ServiceBus (Standard tier) does not accept those messages as the size-limit in Azure ServiceBus is 256KB.

What we see now, is that the message in the rabbitmq queue receives a nack status and that there's constant network traffic over the connection that is used by the shovel. This causes that a lot of bytes are sent by the vessel which affects the available bandwith on the vessel (which is already expensive). We're talking about 150MB of network traffic over 4 hours.

We would expect that the message would be deadlettered in rabbitmq in this case, instead of keeping it in a nack state.

[michaelklishin]

From the description it sounds like the Shovel is not "polling" but trying to re-publish a message it could not successfully publish earlier.

There are several acknowledgement modes to choose from.

Quorum queues support a limit on how many times a delivery can be requeued. Classic queues do not but message TTL can be a good enough alternative.

Queue TTL can be used in combination with dead-lettering, that is, a message that has expired will be deadlettered if their queue has an appropriate argument configured via a policy.

[fgheysels]

Queue TTL can be used in combination with dead-lettering, that is, a message that has expired will be deadlettered if their queue has an appropriate argument configured via a policy.

Does the TTL also apply on messages that have a nack status ?

(Also, I assume you meant Message TTL here instead of Queue TTL ? )

[fgheysels]

From the description it sounds like the Shovel is not "polling" but trying to re-publish a message it could not successfully publish earlier.

There are several acknowledgement modes to choose from.

One of the options we looked at for solving this, was indeed to change the ack-mode and set it to on-publish. However, this would mean in our case that we would loose the message without ever being informed about it.
The message would be send to -in our case- Service Bus. The message would be rejected by SB because it is to big, but I believe it will be acked and be removed from the source queue.

I wonder if it is not possible to deadletter the message, when the shovel tried to deliver it for x times to the target broker (Service Bus), or , if the acknowledgement takes too long ?

[fgheysels]

I also stumbled upon an undocumented feature which would allow to set a max message size: #1812

I think this might also be something that can solve this specific issue.
But, I will also certainly inspect the message TTL in combination with deadlettering.