rabbitmqctl status fails often with badarith on Windows (RabbitMQ 3.9.17)

[PablitoPaul]

Hello,
we use Windows Server 2016 and Windows Server 2019.
Our application uses Erlang OTP 23.1 and RabbitMQ Server 3.8.9 and everything works fine.
Now we've been asked to move to Erlang OTP 23.3.4.14 (or Erlang OTP 24.3.4) and RabbitMQ Server 3.9.16 or 3.9.17.
Our application STILL works fine, and so does the RabbitMQ console at "http://localhost:15672/#/"
but we noticed that when we invoke the "rabbitmqctl status" command (or "rabbitmq-diagnostics status"), we often get the "badarith" error.
It fails randomly... or, more precisely, it succeeds randomly: from the experiments we tried so far, it fails far more often than it succeeds.
The problem occurs on more than one machine.
The problem occurs with both RabbitMQ 3.9.16 and 3.9.17.
The problem occurs with both Windows Server 2016 and Windows Server 2019.

If we ask for the error details, we get this (names have been changed to protect the innocent):

rabbitmq-diagnostics status --print-stacktrace
Status of node rabbit@MYMACHINENAME ...
Stack trace:
** (ArithmeticError) bad argument in arithmetic expression
    (rabbitmqctl 3.8.0-dev) lib/rabbitmq/cli/information_unit.ex:58: RabbitMQ.CLI.InformationUnit.do_convert/2
    (rabbitmqctl 3.8.0-dev) lib/rabbitmq/cli/ctl/commands/status_command.ex:154: RabbitMQ.CLI.Ctl.Commands.StatusCommand.output/2
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:153: RabbitMQCtl.maybe_run_command/3
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:121: anonymous fn/5 in RabbitMQCtl.do_exec_parsed_command/5
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:584: RabbitMQCtl.maybe_with_distribution/3
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:93: RabbitMQCtl.exec_command/2
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:31: RabbitMQCtl.main/1
    (elixir 1.10.4) lib/kernel/cli.ex:124: anonymous fn/3 in Kernel.CLI.exec_fun/2
Error:
:badarith

The error message and stack look similar to the ones of #4734 : that issue was on Linux, and the OP was told to check their "rabbitmq.config" file.
But we use Winodws, and we cannot find that file (there is an "advanced.config" file, but it is empty).

We attach the output of "rabbitmq-diagnostics environment", along with the output of "rabbitmqctl status" from the rare times in which it works
(again, names have been changed to protect the innocent).

Unfortunately, according to our company policies, we're afraid we cannot give you the content of the MNESIA database.
Let un know if there are other diagnostics operations that we can perform to help you analyzing this issue.
Thank you in advance
Paolo and colleagues
rabbitmq-diagnostics environment.txt
rabbitmqctl status.txt

EDIT: We just saw #4900 , fixed in RabbitMQ 3.9.19, published... well, almost today! So early that our Powers That Be have not received their usual notification yet ;) We can try it, if we get permission from the Powers...

[lukebakken]

Yep, it's probably #4900.

Are you seeing any errors in your RabbitMQ log file?

[PablitoPaul]

Hello again @lukebakken and @michaelklishin :

in the logs we see error messages such as:

2022-06-05 22:35:26.808000+02:00 [error] <0.298.0> Command timed out: 'powershell.exe -NoLogo -NoProfile -NonInteractive -Command (Get-PSDrive c).Free'
2022-06-05 22:35:26.808000+02:00 [info] <0.298.0> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',
2022-06-05 22:35:26.808000+02:00 [info] <0.298.0>                                                                                             could_not_determine_disk_free},
2022-06-05 22:35:26.808000+02:00 [info] <0.298.0>                                                                                            12884295680}, retries left: 10

the strange thing is that if we run "powershell.exe -NoLogo -NoProfile -NonInteractive -Command (Get-PSDrive c).Free" manually, it works and it seems "fast". The other strange thing is that the value "12884295680" reported in the log looks like the total RAM, 12 GB, not the free disk space...

About handle.exe, we see only:
[error] <0.643.0> Could not find handle.exe, please install from sysinternals

we have the sysinternals suite on that machine, but in its own directory, since sysinternals is not something that you "install". In the past, we never copied sysinternals binaries in RabbitMQ directories, probably because there has never been the necessity ;)

Meanwhile, we are testing RabbitMQ 3.9.19: so far, it seems to work fine. Yes, most of the time the disk space is reported as "unknown" like Michael said, but our application uses the CLI tool just to check the status of RabbitMQ and not to check the disk space :)

[michaelklishin]

Sounds like status is being abused as a health check which it is not.

[PablitoPaul]

Sounds like status is being abused as a health check which it is not.

Thanks, we will forward your note to our colleagues who maintain the configuration scripts (but status is indeed described in the page you linked as "a common way of sanity checking a node", it's just "slightly more comprehensive" than Stage 1).

[michaelklishin]

Fair enough. That might be a doc leftover, I will see if we should edit it.

[PablitoPaul]

Thanks again, case closed!

[michaelklishin]

This means that free disk space monitoring has failed. We have seen sporadic failures on Windows that we could not explain (the output from handle.exe changes in ways that we could not capture).

So CLI tools will now report free disk space values as "unknown" when this happens.