RabbitMQ service stopped consuming, but remained "healthy" #490
Description
We ran into a case tonight where one of our RabbitMQ services stopped consuming messages without logging any error. This resulted in the queue filling up gradually until we manually restarted the pods from the service which fixed the problem.
Unfortunately, the MessageProcessingHealthCheck did not help here which was supposed to prevent situations like this because it's only checking whether elements are in the internal queue without anything being processed, but the service did not consume messages from its queue any more so the internal queue stayed completely empty.
It was by the way quite easy to inspect the problem thus far thanks to the Motor.NET and RabbitMQ metrics which is nice 😃
But we don't know yet why the consumer apparently stopped working.
For reference, we're using Motor.NET 0.9.10.