Ingress messages were not routed to queues

[sandeepkr977]

Hi Team,

We are using RabbitMQ 3.13.2 in production. Last week we saw some unusual behaviour of Rabbit. Producers were able to produce messages successfully and we were able to see incoming messages flow as well but messages count in [Ready, Unacked, Total] showing 0 in Rabbit. Because of 0 message count consumers were not able consume messages. We had restarted many times consumers but still problem was same. When we have restarted the producer then problem had solved.

I have attached the screenshot for your reference.

[michaelklishin]

Your own screenshots provide evidence of messages being routed since about 15:05. We cannot know what was going on before that.

There are metrics for returned and unroutable messages. A messages that wasn't routed anywhere won't be persisted by definition.

[michaelklishin]

When we have restarted the producer then problem had solved.

That means your topology is set up by the publishers on application boot. If something deletes bindings, for example, all published messages will go nowehere (and at best would be returned, if you publish them as mandatory).

Historically there has been one issue where bindings would disappear but no one could track it down. However, it involved transient queues and quorum queues are never transient. For 4.x with Khepri, we consider it to be solved because bindings are stored very differently.

A few screenshots cannot confirm or deny whether it was the case. It is certainly drastically less frequent than some 5-6 years ago.