Delayed messages reported on metric rabbitmq_global_messages_unroutable_dropped_total

[mdmourao]

Is your feature request related to a problem? Please describe.

We have observed that when publishing messages to an exchange typed x-delayed-message, the RabbitMQ metrics increment the rabbitmq_global_messages_unroutable_dropped_total counter.

While I understand the internal mechanism of the plugin—the exchange receives the message, stores it in Mnesia, and technically does not "route" it to a queue immediately—this behavior has negative observability side effects.

The Problem In a production environment monitored by Grafana (or similar tools), this behavior triggers "Data Loss" or "Unroutable Message" alerts.

Describe the solution you'd like

I would expect this to not be count on the metric: rabbitmq_global_messages_unroutable_dropped_total

Describe alternatives you've considered

Create a new metric to count delayed messages

Additional context

No response

[mdmourao]

Reported by other user here: #36 (comment)

[mdmourao]

Started fork to implement the fix:

mdmourao@673d950

I have zero experience with Erlang, but based on my understanding, the current implementation returns an empty list when a message is delayed. This causes RabbitMQ to interpret the message as 'dropped' (unroutable).

I propose that when a message is delayed, we should signal a successful route to RabbitMQ by returning the exchange name itself. This prevents the broker from flagging the message as unrouted.

I will try to make more tests, is this a fix that makes sense?

[michaelklishin]

@mdmourao this is another known limitation that apparently is absent from the README.

This plugin does not immediately route any messages, it delays them to be routed in the future, and at the time of handling a basic.publish from a client, we cannot know what queues/streams, if any, the message will be routed to.

mdmourao@673d950 does not solve that, it simply returns a non-empty value that will lead RabbitMQ routing to believe that a message was routed while, in fact, it wasn't: it was delayed.

If the goal is to eliminate the "unroutable messages" metric increase that this plugin contributes to, mdmourao@673d950 is enough but technically, with that change your metrics won't be correct and will be distorted differently (by double counting) when the message is actually published. This will be very visible if the delay interval is short, say, 5-10 seconds.

I will have to ask the rest of the core team about this but in any case, this simply shifts the problem around and makes it largely invisible for management UI users when the delay interval is larger than the message rates chart they use.

A fundamental solution would be to change RabbitMQ core to accept a new return value which would mean "a message will be routed later" (so it won't be considered neither routed nor unrouted/dropped), and make this plugin use that return value
as of 4.3.

[michaelklishin]

@lukebakken @the-mikedavis @SimonUnge do you folks have an opinion on the above?

[lukebakken]

I'll take some time to look into the code a bit. Off the top of my head, double-counting published messages seems slightly better than incrementing unrouteable messages.

[michaelklishin]

I'm fine with exploring a core broker modification (for 4.3) that would allow this plugin to avoid both known problematic metric deviations.

[the-mikedavis]

A new return value that says that the routing was "deferred" (or some similar wording) seems reasonable to me 👍

[mdmourao]

I'm happy to contribute, will this be a change on a method? or like a new options flag to indicate "deferred"?

[michaelklishin]

@mdmourao you need to sign the CLA first, then find all the places such as https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_amqp_session.erl#L2556-L2569 (this call site is specific to AMQP 1.0) that increment the metric after assessing what the routing function returned.

We need such routing result handlers to support a new value, deferred (or a comparable value) and handle it for each supported protocol that keeps track of those metrics.

Since the value will be coming from an exchange behavior (interface) implementation, you need to trace what might need to be change there, and because this is the hottest path there is in the entire codebase, we need to be extra certain that the base case scenario (all the built-in exchange types) does not regress using PerfTest and omq.