[Suggestion] Improving visibility of QQ at-least-once dead-lettering failures

[the-mikedavis]

RabbitMQ series

4.1.x

Operating system (distribution) used

macOS

How is RabbitMQ deployed?

Other

What would you like to suggest for a future version of RabbitMQ?

At-least-once dead-lettering for QQs uses an internal queue to guarantee safe dead-lettering. If messages cannot be successfully consumed from the internal dead letter queue, though, they are stuck and the QQ cannot truncate its log past the indexes of the dead-lettered message(s). That can lead to a large disk footprint for reasonably busy QQs, even if the queue appears to be empty in terms of ready messages.

Currently there are some warnings logs that pop up in this situation:

rabbitmq-server/deps/rabbit/src/rabbit_fifo_dlx_worker.erl

Lines 623 to 672 in 522d8fd

	log_missing_dlx_once(#state{exchange_ref = SameDlx,
	logged = #{missing_dlx := SameDlx}} = State) ->
	State;
	log_missing_dlx_once(#state{exchange_ref = DlxResource,
	queue_ref = QueueResource,
	logged = Logged} = State) ->
	?LOG_WARNING("Cannot forward any dead-letter messages from source quorum ~ts because "
	"its configured dead-letter-exchange ~ts does not exist. "
	"Either create the configured dead-letter-exchange or re-configure "
	"the dead-letter-exchange policy for the source quorum queue to prevent "
	"dead-lettered messages from piling up in the source quorum queue. "
	"This message will not be logged again.",
	[rabbit_misc:rs(QueueResource), rabbit_misc:rs(DlxResource)]),
	State#state{logged = maps:put(missing_dlx, DlxResource, Logged)}.
	log_no_route_once(#state{exchange_ref = SameDlx,
	routing_key = SameRoutingKey,
	logged = #{no_route := {SameDlx, SameRoutingKey}}} = State) ->
	State;
	log_no_route_once(#state{queue_ref = QueueResource,
	exchange_ref = DlxResource,
	routing_key = RoutingKey,
	logged = Logged} = State) ->
	?LOG_WARNING("Cannot forward any dead-letter messages from source quorum ~ts "
	"with configured dead-letter-exchange ~ts and configured "
	"dead-letter-routing-key '~ts'. This can happen either if the dead-letter "
	"routing topology is misconfigured (for example no queue bound to "
	"dead-letter-exchange or wrong dead-letter-routing-key configured) or if "
	"non-mirrored classic queues are bound whose host node is down. "
	"Fix this issue to prevent dead-lettered messages from piling up "
	"in the source quorum queue. "
	"This message will not be logged again.",
	[rabbit_misc:rs(QueueResource), rabbit_misc:rs(DlxResource), RoutingKey]),
	State#state{logged = maps:put(no_route, {DlxResource, RoutingKey}, Logged)}.
	log_cycles(Cycles, RoutingKeys, State) ->
	lists:foldl(fun(Cycle, S) -> log_cycle_once(Cycle, RoutingKeys, S) end, State, Cycles).
	log_cycle_once(Queues, _, #state{logged = Logged} = State)
	when is_map_key({cycle, Queues}, Logged) ->
	State;
	log_cycle_once(Queues, RoutingKeys, #state{exchange_ref = DlxResource,
	queue_ref = QueueResource,
	logged = Logged} = State) ->
	?LOG_WARNING("Dead-letter queues cycle detected for source quorum ~ts "
	"with dead-letter exchange ~ts and routing keys ~tp: ~tp "
	"This message will not be logged again.",
	[rabbit_misc:rs(QueueResource), rabbit_misc:rs(DlxResource),
	RoutingKeys, Queues]),
	State#state{logged = maps:put({cycle, Queues}, true, Logged)}.

It can be easy to miss these though since they're only logged once. It would be nice to surface errors like unroutable dead-letters in a more visible way. Maybe the queues page in the management UI could show a warning on these queues for example.

[ansd]

This message is logged once to avoid flooding the logs for each message.
I like the idea to improve the visibility of QQ at-least-once dead-lettering failures.
The simplest approach would be to emit an event to rabbit_event at the same time the log is output.
Another approach could be incrementing some Prometheus metric. Prometheus metrics make it easy to alert on.
The management UI displaying a warning sounds also good to me. When the routing topology is fixed, the warning should disappear.

[michaelklishin]

I really like the idea of starting with a rabbit_event event. We don't really have a lot of events that are "warnings" but different tools can analyze the stream of internal events and react to the presence of such events.