rabbitmqctl status runs into a badarith exception on Linux

[yiannisopit]

Hi all,

After upgrading a 2-node cluster from 3.8.x to 3.9.16 (Erlang 24.3.3), I have the following error in both nodes:

# rabbitmqctl status
Status of node rabbit@StagingMQ1 ...
Error:
:badarith

The same occurs with rabbitmq-diagnostics status.

The services are up and running normally and cluster_status is OK.

The systems are Ubuntu 18.0.4 and installation was done via apt using the packagecloud and erlang-solutions repos.

I found an older issue (#4001) but it regards Windows system.

TIA for your help!

[michaelklishin]

We cannot do much about this with this amount of information.

Running rabbitmq-diagnostics status --print-stacktrace should reveal the stack trace. Something in
your environment is computed to a zero if I had to guess. Server logs (over an hour or two) might
reveal more clues, e.g. maybe the disk monitor is failing and so the % of disk space used cannot
be computed. Something like that.

[pjhavariotis]

Please find below the corresponding stack trace:

# rabbitmq-diagnostics status --print-stacktrace
Status of node rabbit@StagingMQ2 ...
Stack trace:

** (ArithmeticError) bad argument in arithmetic expression
    (rabbitmqctl 3.8.0-dev) lib/rabbitmq/cli/information_unit.ex:58: RabbitMQ.CLI.InformationUnit.do_convert/2
    (rabbitmqctl 3.8.0-dev) lib/rabbitmq/cli/ctl/commands/status_command.ex:154: RabbitMQ.CLI.Ctl.Commands.StatusCommand.output/2
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:153: RabbitMQCtl.maybe_run_command/3
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:121: anonymous fn/5 in RabbitMQCtl.do_exec_parsed_command/5
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:584: RabbitMQCtl.maybe_with_distribution/3
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:93: RabbitMQCtl.exec_command/2
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:31: RabbitMQCtl.main/1
    (elixir 1.10.4) lib/kernel/cli.ex:124: anonymous fn/3 in Kernel.CLI.exec_fun/2

Error:
:badarith

[pjhavariotis]

Additionally, I can see the following errors in the log:

2022-05-05 15:27:07.490822+02:00 [info] <0.349.0> Enabling free disk space monitoring
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>     supervisor: {local,rabbit_disk_monitor_sup}
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>     errorContext: start_error
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>     reason: {function_clause,
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                 [{rabbit_data_coercion,to_list,
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                      [0.7],
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                      [{file,"rabbit_data_coercion.erl"},{line,20}]},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                  {rabbit_resource_monitor_misc,parse_information_unit,1,
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                      [{file,"rabbit_resource_monitor_misc.erl"},{line,18}]},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                  {rabbit_disk_monitor,interpret_limit,1,
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                      [{file,"rabbit_disk_monitor.erl"},{line,324}]},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                  {rabbit_disk_monitor,set_disk_limits,2,
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                      [{file,"rabbit_disk_monitor.erl"},{line,206}]},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                  {rabbit_disk_monitor,enable,1,
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                      [{file,"rabbit_disk_monitor.erl"},{line,358}]},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                  {rabbit_disk_monitor,init,1,
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                      [{file,"rabbit_disk_monitor.erl"},{line,129}]},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                  {gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                  {gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]}]}
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>     offender: [{pid,undefined},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                {id,rabbit_disk_monitor},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                {mfargs,{rabbit_disk_monitor,start_link,[0.7]}},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                {restart_type,{transient,1}},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                {shutdown,300000},
2022-05-05 15:27:07.490938+02:00 [error] <0.348.0>                {child_type,worker}]

[michaelklishin]

What does your rabbitmq.conf or effective configuration look like?

[lukebakken]

Please attach all of your RabbitMQ configuration files.

[pjhavariotis]

StagingMQ1_config.txt
Please find attached the exported configuration by using the RabbitMQ Management API

[pjhavariotis]

rabbitmq_config.txt
Additionally, this is the /etc/rabbitmq/rabbitmq.config file

[pjhavariotis]

Well, it seems that the rabbitmq.config is not correct. The problem was resolved when I removed the aforementioned config file.

[yiannisopit]

It seems that we were using two directives which are obsolete or rather changed nowadays: disk_free_limit and vm_memory_high_watermark without the .relative/.absolute in the end.

Thank you for your time and support!