rabbitmqctl status fails with traceback

[dvarrazzo]

Describe the bug

The commands rabbitmqctl status and rabbitmqctl report return with an error:

$ rabbitmqctl status
Status of node [email protected] ...
Stack trace: 

** (FunctionClauseError) no function clause matching in RabbitMQ.CLI.InformationUnit.convert/2
    (rabbitmqctl 3.12.1) lib/rabbitmq/cli/information_unit.ex:33: RabbitMQ.CLI.InformationUnit.convert({:error, :parse_error}, "gb")
    (rabbitmqctl 3.12.1) lib/rabbitmq/cli/ctl/commands/status_command.ex:152: RabbitMQ.CLI.Ctl.Commands.StatusCommand.output/2
    (rabbitmqctl 3.12.1) lib/rabbitmqctl.ex:174: RabbitMQCtl.maybe_run_command/3
    (rabbitmqctl 3.12.1) lib/rabbitmqctl.ex:142: anonymous fn/5 in RabbitMQCtl.do_exec_parsed_command/5
    (rabbitmqctl 3.12.1) lib/rabbitmqctl.ex:642: RabbitMQCtl.maybe_with_distribution/3
    (rabbitmqctl 3.12.1) lib/rabbitmqctl.ex:107: RabbitMQCtl.exec_command/2
    (rabbitmqctl 3.12.1) lib/rabbitmqctl.ex:41: RabbitMQCtl.main/1
    (elixir 1.15.4) lib/kernel/cli.ex:136: anonymous fn/3 in Kernel.CLI.exec_fun/2

Error:
:function_clause

This happens in a rabbitmq 3.12.12 image deployed using helm from the rabbitmq bitnami chart. As far as I can see, the string gb doesn't appear in any place in the rendered manifest and configuration file, so my polite assumption is that it comes from some default in the code.

The chart was deployed using:

helm upgrade rabbitmq oci://registry-1.docker.io/bitnamicharts/rabbitmq --version 12.6.2 --values ...

where the values files summarize to:

metrics:
  enabled: true

auth:
  username: admin
  existingPasswordSecret: rabbitmq-admin
  existingErlangSecret: rabbitmq-erlang-cookie

replicaCount: 2

memoryHighWatermark:
  enabled: true
  type: absolute
  value: "768Mi"

resources:
  requests:
    memory: "1024Mi"
  limits:
    memory: "1024Mi"

which result in the configuration file:

# rabbitmq.conf

default_user = admin
cluster_name = rabbitmq
cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = rabbitmq-headless
cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.dev.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal

queue_master_locator = min-masters
loopback_users.admin = false

prometheus.tcp.port = 9419
total_memory_available_override_value = 1073741824
vm_memory_high_watermark.absolute = 768Mi

tcp_listen_options.backlog = 128
tcp_listen_options.nodelay = true
tcp_listen_options.linger.on      = true
tcp_listen_options.linger.timeout = 0
tcp_listen_options.keepalive = false

This is probably related to #5083, which didn't follow up with configuration info.

Happy to provide more help, for instance by suggesting how to get the input that crashes the convert() function.

Reproduction steps

Described above.

Expected behavior

command terminating successfully.

Additional context

No response

[michaelklishin]

gb is the default information unit used for formatting.

I cannot reproduce. Something in this specific environment produces a value (an amount of memory available, an amount of free disk space available, this kind of thing) that cannot be converted to gigabytes.

Without knowing what the value is, we cannot do anything.

[michaelklishin]

I recall a few discussions and a change or two in this area, e.g. #5083.

rabbitmqctl status --print-stacktrace

can potentially display more details.

The line in question formats the high VM watermark value which is a configurable value:

    readable_watermark_setting =
      case m[:vm_memory_high_watermark_setting] do
        %{:relative => val} -> "#{val} of available memory"
        # absolute value
        %{:absolute => val} -> "#{IU.convert(val, unit)} #{unit}"
      end

In your configuration file it is set to 768Mi. "Mi" is not an information unit we support.

[michaelklishin]

vm_memory_high_watermark.absolute = 768Mi

reproduces the issue because rabbitmq.conf schema parser does not validate the unit but CLI tools must know how to format it and they don't have a suitable clause.

vm_memory_high_watermark.absolute = 768MB

works perfectly fine.

[michaelklishin]

#10310

[dvarrazzo]

I understand, thank you very much.

The error comes directly from an example in the chart documentation. I guess Mi is accepted for kubernetes, and the chart uses the same value both to configure the k8s resource and to set vm_memory_high_watermark.absolute.

I will raise the issue with the charts maintainers. Thank you very much for looking into it!

[dvarrazzo]

Reported downstream at bitnami/charts#21981

[michaelklishin]

If Mi really is accepted on Kubernetes, I guess we better support it, too.

[michaelklishin]

Yeah, M over MB, Mi over MiB. At least they did not invent their own information units! :D

[michaelklishin]

#10348 has introduced validation of information units in rabbitmq.conf, as well as support for ki, Mi, Gi, Ti, Pi. It will ship in 3.13.0.