Stream content expiration's effects on the Ready message count

[fabiorosa-sn]

Describe the bug

when the rabbit server is using streams, all streams offset are added to the number of "ready" messages. That means, that the number of ready messages will always only increase with time, not event taking the max age into account.

That mixes the stream behavior (offset only increases) with queues behavior (wants to be empty), and makes dashboard and monitoring hard.

Both management-plugin , the prometheus metrics and the RabbitMQ official Grafana dashboard show the same effects: the number of ready messages becomes useless and grafana will always have it in Red as it wants that number to be low.

Reproduction steps

1. enable streams
2. set max-age for any period
3. publish both to queues and streams
   ...

Expected behavior

the count of ready message should at least decrease when max-age for a message is reached and it's not available for consumption anymore.
Ideally, the streams message count and queues message count should not be mixed

Additional context

No response

[michaelklishin]

It's a good question whether there should be separate counters for queues and streams. However, we cannot reproduce the behavior with the metrics: as data is expired, the metric is correctly updated (goes down) in both management UI and the Prometheus endpoint.

Here is what the person who's tried to reproduce observes:

[kjnilsson]

here is the document section which explains how and when stream data is actually deleted: https://www.rabbitmq.com/docs/streams#retention

[fabiorosa-sn]

That is interesting. Here is what I see on my setup:

The same happens to all streams. they are part of a superstream. As you can see, max-age is set to a day. no messages are being sent to this superstream for a week already. The consumers are still connected and the offset lag of the active consumer is 0. Still, all of them still show as 'ready'

We started using superstreams in our system gradually in march ... March 14 we completely replaced the queues for streams for one set of events.. And you can see in the graph that there were never a decrease in the number of ready messages since then.

The server is running rabbitmq 3.12.10 and erlang 25.3.2.7

[acogoluegnes]

The streams may have only one segment file. Retention always leaves at least one segment file, so it may only kick in after more messages get into the stream.

[fabiorosa-sn]

I see. So that makes it even more problematic, from my pov, to have the streams and queues metrics mixed .
I mean, we do have dashboards on specific queues per service, but in general we rely on rabbitmq-overview dashboard to quick asses the current state of things (instead of management ui), but having something like this:

makes it problematic. on a super stream, we have n streams (per partition), that are maybe not that big, but they add to each other... if each stream will not respect the max-age retention depending on its size, then is unpredictable how that will affect the 'ready' metric overall .

[fabiorosa-sn]

Is this question closed ?
Are there any plans to make the behaviour better. Currently, that's quite bad how everything is mixed in a single metric for monitoring any production grade deployment .

[michaelklishin]

@fabiorosa-sn "quite bad" and "make better" are incredibly simplistic problem and solution definitions.

Separating stream and queue metrics may or may not be a good idea, and it would be a major breaking change for monitoring systems.

This is open source software, so you are welcome to look a prototype, a detailed proposal, or something of that kind.

[michaelklishin]

Unless there's obvious interest in separating the metrics.

So far I have seen only 2-3 discussions where no one has even mentioned backwards compatibility, what this might mean for the metric data model, Grafana dashboards, management UI, and so on.

Not much interest and no one is willing to contribute or pay? => Team RabbitMQ will spend our time elsewhere.