Consumer capacity for quorum queues

[olikasg]

Hi team,

We noticed some strange behaviours with consumer capacity metrics with quorum queues.

1. Since 3.10.0 (including latest 3.11), the metric can have only two values: 100% (if any consumers are connected) or 0% (if no consumers are connected).
2. Between 3.8 and 3.9, the same behaviour is observable on the UI, but getting the metric values via the HTTP API shows that it reacts to traffic patterns. If the queue is building up then the metric goes down to 0% or a very small float (0.0003623751791714386), if the consumer is unblocked, then it goes up to 100% (1 or 0.9999999931446886).

For classic queues, the value is much more sensible provided the traffic is fluctuating very near the actual consumer capacity regardless of the RabbitMQ version.

On further investigation it seems that when rabbit_fifo v2 was introduced, the {aux, active} and {aux, inactive} effects are no longer appended which means that

rabbitmq-server/deps/rabbit/src/rabbit_fifo.erl

Line 1089 in fba3329

handle_aux(_RaState, cast, Cmd, #?AUX{capacity = Use0} = Aux0,

is never called and the metric is never updated.

I used the following to test:
Start a rabbit server on the desired version, then use the following perf-test to generate a traffic that fluctuates around the consumer capacity.
For QQs:

java -jar perf-test.jar -x 1 -y 1 -u qq -qq --rate 100 --variable-latency 9990:5 --variable-latency 4000:5 -q 20 -c 2

For classic queues:

java -jar perf-test.jar -x 1 -y 1 -u cl -ad false -f persistent --rate 100 --variable-latency 9990:5 --variable-latency 4000:5 -q 20 -c 2

[kjnilsson]

consumer capacity is one of those things that don't really tell you any more than just looking at the ready messages so it is very likely to just disappear in the future.

[olikasg]

I was thinking along the same line. Looking at the rate of the ready message count. It would be good to add something about the capacity as it is frequently mentioned when diagnosing slow consumers.