How to know when a queue is getting close to prefetch capacity? (_before_ it reaches capacity)

[Victor-N-Suadicani]

TL;DR: How do I auto-scale my service before the queue starts rising?

More details:

For auto-scaling, we are currently examining queue sizes when determining when to scale. Basically we check if any queues in the rabbitmq_queue_messages_ready metric is above 0 - if it's above 0, that means messages are waiting in queue and we need to scale to keep up. When the queue is back down to 0 and has stayed there for a while, we scale down.

However, this is not ideal as scaling is not instant, so the message that is waiting in queue will wait for longer than most of the other messages. Scaling down is also not ideal, as the queue size being 0 could still quickly go to >1 again, if the current consumers are near capacity. So scaling can be "swingy".

What I would prefer is to get some kind of metric that shows the current utilization of the prefetch on a queue across all consumers. For an example, imagine I have a queue and I have 3 consumers taking messages from that queue. Each consumer has a certain prefetch (let's say 8 for the sake of example, but in principle I don't know what the prefetch is ahead of time).

What I would like is to get a percentage of how many unacked messages are in flight compared to the total prefetch capacity. So for my 3 consumers with 8 prefetch each, there can be up to 24 messages in flight before a message would wait in queue. I would like to have a metric that would say 50% when there are 12 messages in flight, 100% when there are 24 messages in flight and 75% for 18 messages in flight, for instance.

If I had such a metric, I imagine I could autoscale much better, as I would be able to scale before I reach capacity (for instance starting the scaling at 75% capacity and downscaling again when at 25% capacity, or something along those lines).

I have tried examining the rabbitmq_queue_consumer_capacity metric, but this seems to be 1 constantly and only falls once a message is queued (i.e. capacity has been exceeded). So this is not an early signal either and doesn't help me more than the rabbitmq_queue_messages_ready metric.

Is there a way to achieve what I want? Keep in mind I don't know the prefetches for all the consumers ahead of time, so this would need to be built into the metric somehow. If it's not possible currently, would it be feasible to add a feature request for this?

I hope this makes sense (feel free to ask follow-up questions) and any advice/help is appreciated! :)

[michaelklishin]

Queues do not have a "prefetch" capacity. Channels do. I cannot think of a client that keeps track of and exposes the number of outstanding deliveries on a channel.

[mkuratczyk]

You can do exactly the same thing with ready messages - don't scale out immediately but rather wait for the ready messages to be above zero for quite some time. Note also that both of these metrics potentially change multiple times a second so in either case, even if see that the metrics is above the threshold for some time, it doesn't mean it doesn't go below. It could be just that your observations happen when it is above

[Victor-N-Suadicani]

If I wait for the ready messages to be above 0 for a certain time period, then I am adding additional waiting time in queue before scaling. That is the opposite of what I want.

My metrics show my ready messages at 0 always and it only goes up when messages are waiting in queue. I want to scale before that happens. Do also check my edit I added to my previous comment.

[mkuratczyk]

I guess for some very specific workloads it could be useful, but I wouldn't expect us to prioritize that anytime soon. If you are willing to submit a PR, I think we could discuss this within the team and decide whether we would approve such PR. If you have no intention of submitting a PR, I'm not sure if there's a point discussing this any further at this point

[Victor-N-Suadicani]

Getting an actual change in was a stretch but I was mostly hoping that there already was some other way to achieve what I was going for :)

But perhaps there isn't? In that case I guess the only option is to keep track of the available prefetch on the client side and then scale based on the messages currently in flight. So I'd need to publish metrics from my consumers saying how much prefetch on the given queue they have and sum that up and then I can use the unacked messages metric from rabbitmq to calculate the metric I'm looking for.

I was just wondering if RabbitMQ provided a better way to do that. Tbh I initially expected the rabbitmq_queue_consumer_capacity metric to be useful in this aspect, but it seems like it doesn't provide any more information than the ready messages metric. So there is no way to get an early warning of rising capacity "natively".

[michaelklishin]

There isn't such "notification" mechanism and this is hardly a commonly requested feature.

If the goal is to then bump the prefetch then you should know that dynamic prefetch is considered to be a bad idea by Team RabbitMQ. It simply never works as well as its proponents claim it will. In fact, it arguably causes more confusion and obscure operational issues for consumers than it solves.

If you need a very high prefetch and you understand the risks, use a high prefetch (say, up to 300 or so, beyond that the gains on throughput with fast consumers will be close to zero).

Client libraries could track the number of unacknowledged deliveries but given that many of them
use thread pool to dispatch to, and basic.ack allows you to acknowledge N messages at a time, including all messages, this will get much more involved than it seems at first.