API shows different status than management UI for queue state

[udiazulayintel]

Describe the bug

Hi,
API shows different status than management UI for queue state, in the below partial response you can see that there is an "idle_since" prop, but state = "running".
In the management UI - it shows "idle" correctly.

Please advise.

"head_message_timestamp": null,
"idle_since": "2023-08-06T21:59:42.043+03:00",
"memory": 777064,
"message_bytes": 160051171,
"message_bytes_paged_out": 0,
"message_bytes_persistent": 160051171,
"message_bytes_ram": 225,
"message_bytes_ready": 160051171,
"message_bytes_unacknowledged": 0,
"message_stats": {
"publish": 2601590,
"publish_details": {
"rate": 0.0
}
},
"messages": 583248,
"messages_details": {
"rate": 0.0
},
"messages_paged_out": 0,
"messages_persistent": 583248,
"messages_ram": 1,
"messages_ready": 583248,
"messages_ready_details": {
"rate": 0.0
},
"messages_ready_ram": 1,
"messages_unacknowledged": 0,
"messages_unacknowledged_details": {
"rate": 0.0
},
"messages_unacknowledged_ram": 0,
"name": "q.testerName_testerSite_1",
"node": "rabbit@HAGAPP1007",
"operator_policy": null,
"policy": null,
"recoverable_slaves": null,
"reductions": 1639929095,
"reductions_details": {
"rate": 0.0
},
"single_active_consumer_tag": null,
"state": "running",
"type": "classic",
"vhost": "prime-idc"

Reproduction steps

1. queue with idle state
2. run /api/queues/
3. check state
   ...

Expected behavior

queue state should be idle

Additional context

No response

[michaelklishin]

Management UI uses exactly the same HTTP API. There is no "special internal API" it relies on.

Querying the API likely results in queue processes performing some activity (this is possible even though stats samples are collected periodically). Or it's a timing issue between the requests.

If you plan on using the queue state metric for consumer activity monitoring, don't. It is not meant
to be used that way and simply indicates queue leader replica's Erlang process activity, which is
not necessarily related to application activity.

When it does matter is when the metric reports that the queue is failing. All other states are not particularly interesting. Use numerical consumer metrics with a reasonable threshold. Since the metrics
such as delivery rate are computed over a window of time, they won't be perfectly precise at any given moment but when averaged out, will give you a decent idea of queue activity.

idle_since is another metric I wish was not exposed publicly. It tells you when the queue leader replica's Erlang process last entered hibernation, a runtime detail no one but the contributors should pay attention to 99.9% of the time.

[udiazulayintel]

Ok, Thanks for the detailed info.

Can i rely on "Number of consumers" for a queue? as i did not find a way to check it in RabbitMQ.Client .net lib.

Thanks!
Udi

[michaelklishin]

Yes, you can. And in modern versions, on their activity status, their channel prefetch (this would need aggregating with channel metrics, which can be too expensive to be worth it) and even consumer capacity.

Note that more complex function that determine consumer activity usually do not work well. Which is why I suggested a single numerical metric: the delivery rate. It can be zero at times, but ultimately there is no better indicator.

Consumer count is also a reliable metric but as you know, having a consumer registered does not mean that the consumer is doing anything productive. It might as well be stuck for any reason. Which is why modern RabbitMQ versions enforce a delivery acknowledgement timeout.

[udiazulayintel]

Thanks much!