Unusual ingress message rate values reported from GET /api/queues/{name}

[laithalissa]

Describe the bug

I'm experiencing some alarming inconsistencies with the queue message rate metrics from the management API which I'm depending on very heavily.

I'm finding that rate is occasionally reported as an incorrect value, orders of magnitude more than it really is.

To demonstrate, I submitted 3 messages, separated by a few seconds, to an exchange to which a queue is subscribed.
I have a while loop hitting the endpoint to monitor the publish stats over this time period.

http://localhost:15672/api/queues/%2F/myQueue' \
| jq '.message_stats.publish_details'

This is the output- I've removed the duplicate output to keep it readable

2023-05-30 16:35:35 {"rate":0}
  // [21s] No change in this time period
2023-05-30 16:35:56 {"rate":0}
2023-05-30 16:35:56 {"rate":0.2}
  // [ 8s] No change in this time period
2023-05-30 16:36:04 {"rate":0.2}
2023-05-30 16:36:04 {"rate":0}
  // [ 8s] No change in this time period
2023-05-30 16:36:12 {"rate":0}
2023-05-30 16:36:12 {"rate":740}
  // [ 7s ] No change in this time period
2023-05-30 16:36:19 {"rate":740}
2023-05-30 16:36:20 {"rate":0}

For the first message, the rate didn't change, it remained zero
For the second message, I saw an 8 second period of rate: 0.2
For the third message, I saw a 7 second period of rate: 740

Here is a screencap of the rabbit ui for this queue, capturing this activity.

RabbitMQ 3.11.15
Erlang 25.3.2
Deployed on kubernetes using the bitnami rabbit-cluster-operator, version, chart version 3.3.2
Exchange is of topic type, queue is durable, quorum type. There's nothing else using the exchange or queue when I test.

Reproduction steps

This issue doesn't reproduce consistently

Assuming the setup in my description:

1. Create exchange of type topic
2. Create durable quorum queue, add binding to topic
3. Monitor the /api/queues/%2F/myQueue endpoint for metrics (.message_stats.publish_details)
4. Submit a single message to the queue
5. Repeat many times until you see the rate spike to an usually high value

Expected behavior

rate should be a value which reasonably reflect the rate of messages

Additional context

I've reproduced this issue about 7 times (I ran approx 50 tests), here are the unique rate values I yielded, the 0.4, 0.6 values had 2-3 messages submitted at once, respectively, so they can be ignored

"rate":0
"rate":0.2
"rate":0.4
"rate":0.6
"rate":730.4
"rate":731
"rate":731.2
"rate":734.8
"rate":740
"rate":743.4

[michaelklishin]

I'm afraid that's not enough information to suggest anything specific. The commended monitoring solution is Prometheus (or a compatible tool), to the point
that management UI long has had a way to disable metrics entirely.

Metric samples are emitted periodically. HTTP API then aggregates them across all nodes.

The chart does not contain any values that would be in the 700+ range. My best guess is that some
sliding window calculations smoothen the irregularity.

Assuming that you use this for backlog monitoring, a single reported value is not going to be useful in practice. It would make you depend too much on the specific moment the value is retrieved.
Given that the rates are computed over a sliding window of samples, they are never perfectly accurate.

[laithalissa]

@Zerpet to give you a bit of context, this queue is effectively a buffer for real time requests. It takes about 5-8s to process a request, but when we get an unexpected increase of messages, we're unable to keep up with our target "sla" of under 10s. Scaling workers is a very slow process, it can take 3-5 minutes for a consumer to be ready after a scale event. As such I want to scale at the earliest possible time. I'm attempting to implement this with 2 keda scalers. One to trigger a scale for when the message rate exceeds a threshold, and a second (backup) scaler which scales when there's a backlog of message. The second scaler is an emergency response for when we're exceeding our anticipated growth (or something changes regarding processing time). Due to the unavoidable initialisation times, I'm not able to accomodate any delay in the scaling responses, hence I'm always aiming for 0 ready messages, and am very reluctant to add a potentially large scrapeInterval delay in my solution.

In order to make it work with prometheus, I would need to reconfgiure prometheus with a sufficiently low enough global scrape interval, in the order of a few seconds, which isn't ideal. Granted it's not much different than having Keda repeatidly poll the rabbitmq http api, but I feel it's compromising my general-purpose monitoring tool just to workaround this issue, since prometheus is the subject to managing much higher resolution data, and I have to be careful regarding other service monitors going forward to ensure they don't flood prometheus with low interval metrics.

The options as I see them are:

1. Suck it up and use prometheus with a very low scraping interval
2. Run a second prometheus as a dedidcated shim between rabbit and keda
3. Write a proxy to handle this for me (keda external scaler)
4. Implement/wait for Keda team to implement a smarter http call to rabbit

@michaelklishin you put a negative spin on the HTTP api previously, is there a plan/thought to deprecate and remove it in favour of the promtheus metrics endpoint?

[michaelklishin]

I only meant that for metrics, there is a better option: Prometheus. It's been possible to disable metrics in the HTTP API for several years.

With everything else that the HTTP API provides, it does a good job, and there are no plans to deprecate the entire API. It does not seem to be realistic for us to remove the metrics part entirely, we cannot require every user to always have a Prometheus installation around.

For production systems, the recommendation is to use Prometheus for a variety of reasons. Long term
metric storage is one good example.

[Zerpet]

Thanks for the context! It's an interesting use case. I agree with the issue you reported in Keda kedacore/keda#4507. If that doesn't happen soon-ish, you could try to amqpprox by Bloomberg for a proxy and Keda external scaler (option 3). Option 2 also sounds good to me, in combination with a Prometheus Scaler, could do the trick.

Regarding the inaccurate metric coming from the HTTP API, our focus is on the Prometheus endpoint. That being said, we'll definitely welcome a PR, although I think it'll be best to invest on kedacore/keda#4507

A long shot, have you tried with 3.12.0-rc.4 as Michael tried out? DockerHub should already have the RC images built.

[laithalissa]

@Zerpet Thanks for the suggestions.

I haven't tried 3.12.0-rc.4 yet, I didn't want to introduce unknowns from an RC, but maybe I should since you're scheduled for a stable release soon. Trouble is it's not that easy to reproduce. I had been using this version for ~4 weeks when I suddenly got alerted that the cloud provider span up a 400 instances for one request 💀 . Fun times.

@michaelklishin thank you for clarifying.

I'll followup with any findings from 3.12.0-rc.4.

[laithalissa]

Reproduced the issue on 3.12.0-rc.4. The rate isn't as high as it was previously.

I found a sweet-spot of leaving 5 minutes between requests (while true; do qmsg; sleep 300; done;), yielding {"rate":47.2} and {"rate":46}, so far.

[michaelklishin]

Given a quorum queue named qq.1 on a 3.12.0-rc.4, I publish a message per second using Bunny in a loop:

c = Bunny.new; c.start
ch = c.create_channel
qq = ch.quorum_queue("qq.1")

loop { qq.publish(""); sleep 1 }

and I monitor the rate using

for i in {1..1000}; do curl -X GET -s -u guest:guest http://localhost:15672/api/queues/%2F/qq.1 | jq ".message_stats.publish_details"; sleep 1; done

The output is

null
null
null
null
null
{
  "rate": 0
}
{
  "rate": 0
}
{
  "rate": 0
}
{
  "rate": 0
}
{
  "rate": 0.6
}
{
  "rate": 0.6
}
{
  "rate": 0.6
}
{
  "rate": 0.6
}
{
  "rate": 0.6
}
{
  "rate": 0
}
{
  "rate": 0
}
{
  "rate": 0
}
{
  "rate": 0
}
{
  "rate": 0
}
{
  "rate": 0.2
}
{
  "rate": 0.2
}
{
  "rate": 0.2
}
{
  "rate": 0.2
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}
{
  "rate": 1
}

[michaelklishin]

When I change the publishing rate to be about 5 messages per second

loop { qq.publish(""); sleep 0.2 }

I get the following:

{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 4.8
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}
{
  "rate": 5
}

[mkuratczyk]

I've also tried with different message rates, different number of publishers and some other modifications and never seen a value higher than expected.

[laithalissa]

Interesting, I've reproduced it in 2 different versions of rabbitmq in a production environment. I can maybe try to repro in a local setup with docker compose or something. What testing setup are you using for this? Bear in mind I need to simulate an HA rabbit cluster setup with a quorem queue