Free disk space monitor computes (or returns) the available disk space as 'NaN'

[GuillaumeCisco]

Following the discussion #5143 (comment)_

There is an issue with the disk_space metrics returning 'NaN' (wrapped in quotes) making the whole prometheus experience broken.

By the way, I find totally disrespectful what you did @michaelklishin.
On a github discussion, GUESS WHAT? People are here to discuss.
Nice people tried to understand what is going on, and find a solution.
Helping others.
I repeat: helping others.

[michaelklishin]

@GuillaumeCisco you did not provide any details (log files, effective node configuration), so it's very difficult to help you.

Then you hijacked an existing discussion, which is not acceptable for this repository (if it is for other maintainers, good for them). Every project has certain guidelines and I'm afraid you do not get to dictate them, especially when you are looking for help from the maintainers for free.

Instead of repeating the same thing over and over, please provide log files from the affected node and its effective configuration.

[GuillaumeCisco]

@michalklishin, what the purpose of creating a new discussion for the exact same problem which has never been resolved in the first place?
I hijacked nothing, please behave correctly. What do you expect from others if you unilaterally decide to close an active and interesting discussion? Where people think and try to understand what's going on?

I will be delighted to provide logfiles, actually this is what in the first place was trying to achieve. How to capture logs and which ones to send? Should I filter things?
I'm not working everyday on rabbitmq like you do, so please, bear with. You cannot expect me to be the perfect submitter in this project a first try, I also have priority and things to do.

[michaelklishin]

@GuillaumeCisco looks like you are not interested in finding a solution, or at least understanding the root cause of the issue, you just need to blow some steam off.

Go do it somewhere else.

[michaelklishin]

@GuillaumeCisco a lot of maintainers, and on a lot of forums, hijacking an existing discussion is considered a violation of community rules. If you don't like it, you can set your own rules for your own repositories or communities.

[michaelklishin]

I'm not sure why some people believe they are entitled to free support and it is OK to demand things from open source software maintainers.

Please provide the information we asked for and follow the rules, or resolve the issue on your own. I repeat: no open source maintainer owes you anything.

[michaelklishin]

Relevant information from an earlier discussion:

• RabbitMQ version is 3.11.11
• The environment is a flavor of Kubernetes
• rabbitmq_disk_space_available_bytes is the metric that ends up being computed or rendered as 'NaN'
• A potential "permission issue" was mentioned without any specifics, so the free disk space monitor may be failing with an exception (that will be visible in the logs)

[GuillaumeCisco]

"permission issue" may be a clue.
rabbitmq cluster is running on a GKE autopiltot environment i.e nodes are managed automatically.
Just posting this piece of information, if it can ring a bell.

[michaelklishin]

This line in rabbit_disk_monitor returns an atom 'NaN' when it cannot read the key from ETS. Returning a numeric value would let Prometheus scrapers parse the output but using 0 (one of the values suggested) would immediately raise a resource alarm on the node that will never clear.

In turn, that will block all publishers across the entire cluster. I doubt this is a better outcome.
What other value can be safely returned and why is a good question.

[michaelklishin]

Prometheus metric documentation does not mention what are the acceptable "no value" values are for gauges or counters, in other words, how to express None (as in Option<T>) or null. 0 will not work for this specific metric for the aforementioned reason.

@ikvmw alternatively we can avoid emitting the metric if it is NaN.

[michaelklishin]

Looks like not emitting the metric will be quite difficult because rabbit_mgmt_external_stats explicitly tries to compute it.

Then it also returns disk_free_monitoring_disabled, a special value filtered out by the management UI. So perhaps we can return the same value and consider filtering it out in more places.

[michaelklishin]

CLI tools filter out NaN in rabbitmq-diagnostics status output. So if we were to change this value, we'd have to update CLI tools accordingly.

[lukebakken]

Please provide log files from every node in your cluster. We're looking for the string Free disk space monitor encountered an error -

https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_disk_monitor.erl

[michaelklishin]

The original poster won't provide anything since they are now banned org-wide, including a duplicate account.

But we do know that NaN is only ever returned when the monitor cannot compute a value. We just need to decide if there may be a "no value" value that would work for Prometheus scrapers or other tools that consume metrics in the same format.

[michaelklishin]

Some more findings by other team members:

• An omitted value is handled by Grafana as "no data", which is correct
• undefined is not handled by at least some tools
• null null is not handled by at least some tools

Modifying the Prometheus plugin to leave the metric out seems to be the only practical solution.

[michaelklishin]

By exhausting all obvious (to a few members of our team) options, we've so far arrived at #8740, with enough details and understanding of various options available to move it to an issue.

[michaelklishin]

#8740 is now resolved, the change will ship in 3.12.2 and 3.11.20.