Expose publish reject counter via api and metrics and log in server logs

[zaeemarshad]

Is your feature request related to a problem? Please describe.

When max-length is enabled on quorum queues (and probably Streams but not tested) with overflow set to reject-publish, new messages are rejected. This rejection is notified to the clients and is up to the clients to handle them. As a server operator, there's no mechanism to know when a queue has reached max-length and is rejecting publishes

Describe the solution you'd like

As a server operator, I'd like the following:

• Have a log message emitted in server logs when publish operations are rejected due to max-length
• Have a counter per queue which is incremented every time a publish is rejected due to max-length. This counter should be exposed via the api as well as the prometheus metrics.
• Expose the max-length/max-bytes as a prometheus metric on a per-queue basis

As an operator, I can then combine the above two in an alert that can notify us when a queue is approaching or has hit the max limit and intervention is required.

Describe alternatives you've considered

No response

Additional context

No response

[michaelklishin]

In general, if something is up to the client, it is up to the client, including when it comes to observability.

RabbitMQ does not log anything related to individual messages, doing so on a busy system is a no-go as it will overwhelm the logger, waste resources, and have operational consequences.

It also does not keep track of totals published, confirmed or rejected, only a rate in the latest metric collection time window. Adding a metric like this makes sense if it can be collected without significant changes.

Max length and max bytes are already available in management UI thanks to the fact that effective queue x-arg definition is available. This is also an operator-configurable value, and it can be forcefully set via an operator policy. Exposing what you set via a metric does not sound very convincing to me.

RabbitMQ cannot address every issue in poorly written applications. When, for example, a channel is closed with pending confirms, we log it because 1. The client won't have any way to observe this internal metric, and 2. This is a relatively rare event (compared to the messages published/second rate, and in systems with low channel churn anyway). But we cannot log messages for every occurence of an event that may indicate that a consumer must do something.

[zaeemarshad]

It also does not keep track of totals published, confirmed or rejected, only a rate in the latest metric collection time window. Adding a metric like this makes sense if it can be collected without significant changes.

I am thinking of a counter per-queue that is incremented whenever a publish is rejected. It does not need to account for the rate of messages. The thought here is that monitoring tools like Prometheus can use rate on the counter itself. I am not sure if this requires a significant change or not.

Max length and max bytes are already available in management UI thanks to the fact that effective queue x-arg definition is available. This is also an operator-configurable value, and it can be forcefully set via an operator policy. Exposing what you set via a metric does not sound very convincing to me.

On second thought, I agree with you. I don't think all config elements should be exposed via metrics. This is something that I can handle on my own i.e. by exposing the runtime values as a metric using the Prometheus textfile collector. This can then be fed into an alert which compares the the message count of the queue against this limit and fires an alert.

RabbitMQ cannot address every issue in poorly written applications.

Fair and it is expected not to. However, Operators do not think of applications in isolation. We look at that from a systems perspective and generally think of how we can have monitoring, alerting and guard-rails in place to ensure that if there's a misbehaving application, we can get in front of the problem before it becomes a major issue.

Consider a topic exchange with two queues A and B bound to it. There's a single Publisher P while consumer A and consumer B are bound to queues A and B respectively and an operator policy that defines the max-length for both queues. If consumer B suffers an issue (e.g. some upstream being unavailable) where it continuously NACK/requeues messages to a queue and a decently high incoming volume, we can quickly get into a situation where queue B will hit the max-length and stop accepting new messages while queue A continues to handle the messages.

If the issue disappears after some time, we will be in a situation where queue B will have potentially lost some messages and there would have been no visibility when something like this had happened. An obvious workaround is to have a DLX queue with reject-publish-dlx set as the operator policy but it brings its own set of complications. Just my thoughts on how we handle and run things.

[mkuratczyk]

This is something that I can handle on my own i.e. by exposing the runtime values as a metric using the Prometheus textfile
collector. This can then be fed into an alert which compares the the message count of the queue against this limit and fires an alert.

Indeed, you even have the predict_linear PromQL function to estimate when this could occur and alert appropriately.

queue B will have potentially lost some messages and there would have been no visibility when something like this had happened.

if you set the mandatory flag when publishing, the rejected message is returned. Therefore your publisher knows immediately that something's wrong, it can alert you, stop publishing or whatever is adequate.

I get your overall point, but as Michael said, we keep getting requests to fill the gaps in application monitoring and we already have so many metrics, that it's a problem on our side and for some other users (case in point - we are about to reduce the number of metrics exposed: #9627). I feel like a potential compromise could be a single global metric. There's currently rabbitmq_global_messages_dead_lettered_rejected_total, we could add rabbitmq_global_messages_overflow_rejected_total or something like that - then the number of metrics doesn't increase significantly (we have users with hundreds of thousands of queues, so anything per-queue can be a problem for someone else), while it gives you just enough to alert on and start digging deeper. It wouldn't tell you which queue is full, but would tell you that at least one is.

[michaelklishin]

A single global counter similar to rabbitmq_global_messages_dead_lettered_rejected_total in purpose sounds good to me

[zaeemarshad]

A single global counter is a good option. We can feed it into an alert coupled with other monitoring signals to prevent problems.

[michaelklishin]

Also, if you think about it, the max-length and the overflow behavior are usually operator-defined. If your system does not assume that this can happen, and applications are not ready to handle it, or you cannot afford rejected messages, perhaps those settings should not be used, or should not be used with a low limit value.

reject-publish very explicitly suggests right in the name that it may or may not be safe, and max-length is a guardrail.

[zaeemarshad]

Consider the use case of multiple queues in a single cluster. From an operator perspective, it is possible that consumers can suffer an issue and stop consuming or NACK/Requeue. In this case memory usage will grow and can quickly get to a point where the whole cluster is destabilised i.e. the rabbit process gets OOM killed repeatedly on restarts until memory is increased on the system. To prevent this from happening, it makes sense to have the max-length guard rail. However, we also want to know if we are getting close to that point or have hit that once or twice and intervene to prevent further system instability.