"/metrics" endpoint request failed with "strconv.ParseFloat: parsing "unknown": invalid syntax" error

[Arun-N]

Describe the bug

RabbitMQ-Overview Grafana dashboard was showing no data today. When I checked the targets on Prometheus, http://10.2.0.74:15692/metrics endpoint was down (see screenshot below)

To Reproduce

Steps to reproduce the behavior: No steps.

Expected behavior
http://10.2.0.74:15692/metrics endpoint should be up

Screenshots

Version and environment information

• RabbitMQ: 3.10.2-management
• RabbitMQ Cluster Operator: 1.14.0
• Kubernetes: 1.23
• Cloud provider: Linode (LKE)

Additional context

• Everything was working fine yesterday.
• http://10.2.0.74:15692/metrics/detailed endpoint is working fine (1/1 up). Only http://10.2.0.74:15692/metrics is down

[mkuratczyk]

Seems like some metric returned undefined as the value (which should be a float number). Can you curl 10.2.0.74:15672/metrics and share the result with us, so that we know which metric that is?

[michaelklishin]

strconv.ParseFloat: parsing "unknown" is a message from a C# client of sorts, not RabbitMQ.

[mkuratczyk]

The error is coming from Prometheus itself (a Go error), because RabbitMQ returns the value undefined, which doesn't meet the Prometheus metrics format specification:

https://prometheus.io/docs/instrumenting/exposition_formats/#comments-help-text-and-type-information

value is a float represented as required by Go's ParseFloat() function. In addition to standard numerical values, NaN, +Inf, and -Inf are valid values representing not a number, positive infinity, and negative infinity, respectively.

[Arun-N]

Can you curl 10.2.0.74:15672/metrics and share the result with us, so that we know which metric that is?

Unfortunately I had to get it resolved, so I deleted & re-created the cluster which seems to have solved the issue - at least for now. If it happens again, I'll send output of the above. Or should I send it now anyways?

[mkuratczyk]

If it works, there is no point. There must be a situation where a metric is returned as undefined - perhaps very early after the deployment, when it was never incremented or something like that, which leads to this error on the Prometheus side, since a numerical value is expected. But without knowing which metric that was, it's hard to find. If it happens again - please curl and share. Thanks

[ElfoLiNk]

Hi @mkuratczyk in my case was:

rabbitmq_disk_space_available_bytes unknown

[lukebakken]

@ElfoLiNk @Arun-N

#5726 should fix this issue, since it changes from using unknown to NaN when the free disk space can't be determined.

[GuillaumeCisco]

I just got the issue:

strconv.ParseFloat: parsing "'NaN'": invalid syntax while parsing: "rabbitmq_disk_space_available_bytes 'NaN'"

With prometheus:v2.45.0 and rabbitmq:3.11.11-management docker images.
Is it a real issue? Does it block other metrics?
What should I do?

Is there a way to disable the rabbitmq_disk_space_available_bytes metrics? A way to correctly configure it?
I use kubernetes and pvc for having a persistent volume of 20Gi for rabbitmq.

EDIT: I can see in the returned metrics:

# TYPE rabbitmq_disk_space_available_bytes gauge
# HELP rabbitmq_disk_space_available_bytes Disk space available in bytes
rabbitmq_disk_space_available_bytes 'NaN'

running a kubectl -n <namespace> exec <rabbitmq-pod> -- df -ah tells me only 2% of the 20Gi PVC is used...

/dev/sdb         20G  222M   20G   2% /var/lib/rabbitmq

EDIT2: My kubernetes cluster in on GKE autopilot, where nodes are automatically managed, could it be a clue?
I get a NaN value because the permission is forbidden?

[ikavgo]

I think the problem is that NaN is in quotes

[GuillaumeCisco]

You mean this code?
https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_disk_monitor.erl#L106
How could we tell to get "NaN" and not "'NaN'" ?

[GuillaumeCisco]

By the way, shouldn't I get a real value?
As the disk space is 2% filled?

[lukebakken]

For some reason RabbitMQ can't compute the free disk space in your environment. It uses the df command.

[GuillaumeCisco]

Just tested this code on go playground: https://go.dev/play/

package main

import (
	"fmt"
	"strconv"
)

func main() {
	fmt.Println(strconv.ParseFloat("NaN", 64))
	fmt.Println(strconv.ParseFloat("'NaN'", 64))
}

Same error with double quotes + quotes.

[ikavgo]

Luke you are right, we shouldn't return NaN (what you meant by javascript reference?), however when ets table isn't populated metric shouldn't be returned at all.

@GuillaumeCisco what you mean by real value? the value is cached periodically, you shouldn't have this value present always, IIUC.

[GuillaumeCisco]

I mean, why do I still get a NaN?
Shouldn't I get a real value, even after hours after my rabbitmq cluster is deployed?

[michaelklishin]

It depends on what metric specifically we are talking. Metrics are not guaranteed to be present. If you don't have any connections, or a channel does not have any consumers, different metrics won't be defined.

Free disk space monitoring is an optional metric. Depending on the OS and deployment properties, may or may not succeed. On Windows, for example, it relies on a 3rd party tool being available in PATH.

See server logs for clues about free disk space monitor erroring or terminating.

[GuillaumeCisco]

I have nothing in the logs regarding this metrics :/

[lukebakken]

(what you meant by javascript reference?)

I didn't realize NaN isn't a first-class thing in Javascript.

[GuillaumeCisco]

Then, should we simply modify the code for returning 0 instead of 'NaN'?

[ikavgo]

No, NaN is a result of computation. If disk metric doesn't have a value it just doesn't exist and shouldn't be rendered.

[GuillaumeCisco]

Ok, so how can disable this metrics if we don't get any value?

[michaelklishin]

Free disk space monitor disables itself if it cannot compute a value.

rabbitmqctl eval 'rabbit_disk_monitor:set_enabled(false).'

will manually invoke the same function and thus disable it.

[michaelklishin]

We do not guess in this community. If you are looking for more free help from the core team, please share logs and your node's configuration.

[michaelklishin]

In fact, since the original discussion was answered and then another person hijacked it to ask another question, I am going to close and lock this as resolved.

If someone wants to ask a question, they should start a new discussion and provide effective node configuration plus logs. We will not guess what exactly is going on with the disk space monitor, in particular when the version is question is less than one month away from going out of community support.

[michaelklishin]

This ultimately led to #8740, which is now resolved and will ship 3.12.2 and 3.11.20.

[michaelklishin]

Another outcome of the story: if a discussion is closed and/or answered, it is closed, please start another one (links between discussions work perfectly fine on GitHub).

If you don't do that, the team will close the discussion. When that happens, don't attack the team and try to impose your own rules around here, you don't get to decide how other people use Discussions and Issues in their projects.