RabbitMQ calculates available memory incorrectly

[Regmir]

Hi, we are deploing RabbiMQ 3.10.6 alpine docker to 1.24 Kubernetes with 0.6Gi RAM, but in logs we got
Memory high watermark set to 9579 MiB (10044397977 bytes) of 23947 MiB (25110994944 bytes) total
Management UI also displays incorrect value. vm_memory_calculation_strategy = allocated, but tried rss and legacy too. Please help, is this a bug or we doing something wrong?

[mkuratczyk]

You are doing at least one thing wrong: you are deploying a 6-year old version of RabbitMQ. More importantly in this case, Erlang is also ~6 years old and I'm pretty sure Erlang was unaware of cgroups back then. Check with a contemporary version.

[Regmir]

Sorry, Imade a typo, we deployed 3.10.6 version

[mkuratczyk]

Are you using the Operator? Can you check the defaults deployment? For me it looks good:

$ cat test.yaml
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: test
spec:
  replicas: 1
  resources:
    requests:
      cpu: 1
      memory: 600Mi
    limits:
      cpu: 1
      memory: 600Mi

$ kubectl apply -f test.yaml
$ kubectl logs test-server-0 | rg watermark
2023-03-20 11:06:56.845506+00:00 [info] <0.287.0> Memory high watermark set to 192 MiB (201326592 bytes) of 480 MiB (503316480 bytes) total

This is on GKE 1.26 but we have users on many Kubernetes flavours and versions and I don't recall such reports.

[Regmir]

We do not use RabbitMQ operator, maybe we should use total_memory_available_override_value to define memory value?

[mkuratczyk]

It's a workaround if you want, but it shouldn't be needed. In the operator, we use it to lower the memory used by calculations from the container limits, because Erlang can temporarily use a bit more than it was assigned, so we are reducing the risk of an OOMkill (that's why you see "of 480 MiB", even though the definition says "600Mi".

I'd recommend using the operator or at least giving it a try to see if the issue is still there. That could help us narrow down what's different about your configuraiton.

[Regmir]

Thank you for help

[carlosjgp]

I'm seeing a similar problem with a newer version 3.10.13 published in December'22

|Home(H)|Network(N)|System(S)|Ets(E)/Mnesia(M)|App(A)|Doc(D)|Plugin(P)Interval: 1500ms                                 | 3Days 1:47:35    |
|Id |App                            |ProcessCount(p)     |Memory(m)           |Reductions(r)    |MsgQ(mq)  |Status      |version          |
|1  | rabbit                        | 209                | 2.3876 GB          | 46794168        | 0        | Started    | 3.10.13         |
|2  | ra                            | 111                | 757.1649 MB        | 28123620        | 0        | Started    | 2.4.5           |
|3  | rabbitmq_web_dispatch         | 59                 | 10.1428 MB         | 931936          | 0        | Started    | 3.10.13         |
|4  | kernel                        | 54                 | 141.6303 MB        | 1656580         | 0        | Started    | 8.5.2           |
|5  | rabbitmq_management_agent     | 39                 | 1.6813 GB          | 2943356         | 0        | Started    | 3.10.13         |
|6  | ssl                           | 20                 | 204.1230 KB        | 131352          | 0        | Started    | 10.8.6          |
...

$ rabbitmq-diagnostics status
Status of node [email protected] ...
Runtime

OS PID: 11
OS: Linux
Uptime (seconds): 267229
Is under maintenance?: false
RabbitMQ version: 3.10.13
RabbitMQ release series support status: supported
Node name: [email protected]
Erlang configuration: Erlang/OTP 25 [erts-13.1.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 1.1.1s  1 Nov 2022
Erlang processes: 604 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /operator/enabled_plugins
Enabled plugins:

 * rabbitmq_prometheus
 * rabbitmq_shovel_management
 * rabbitmq_peer_discovery_k8s
 * accept
 * rabbitmq_peer_discovery_common
 * rabbitmq_shovel
 * amqp10_client
 * prometheus
 * rabbitmq_management
 * amqp_client
 * rabbitmq_web_dispatch
 * cowboy
 * cowlib
 * rabbitmq_management_agent

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/[email protected]
Raft data directory: /var/lib/rabbitmq/mnesia/[email protected]/quorum/[email protected]

Config files

 * /etc/rabbitmq/conf.d/10-defaults.conf
 * /etc/rabbitmq/conf.d/10-operatorDefaults.conf
 * /etc/rabbitmq/conf.d/11-default_user.conf
 * /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf

Log file(s)

 * /var/log/rabbitmq/[email protected]_upgrade.log
 * <stdout>

Alarms

(none)

Memory

Total memory used: 0.5179 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 0.6872 gb

binary: 0.1825 gb (35.24 %)
quorum_ets: 0.1031 gb (19.91 %)
reserved_unallocated: 0.0559 gb (10.79 %)
queue_procs: 0.0431 gb (8.32 %)
code: 0.0358 gb (6.9 %)
quorum_queue_procs: 0.0249 gb (4.82 %)
other_proc: 0.0239 gb (4.61 %)
other_system: 0.0178 gb (3.43 %)
allocated_unused: 0.0163 gb (3.16 %)
other_ets: 0.0054 gb (1.04 %)
plugins: 0.0034 gb (0.65 %)
mgmt_db: 0.0015 gb (0.29 %)
atom: 0.0015 gb (0.28 %)
metrics: 0.0012 gb (0.22 %)
connection_other: 0.0009 gb (0.16 %)
mnesia: 0.0005 gb (0.1 %)
connection_readers: 0.0003 gb (0.05 %)
connection_channels: 0.0001 gb (0.01 %)
connection_writers: 0.0 gb (0.01 %)
msg_index: 0.0 gb (0.01 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 137, limit: 65439
Sockets: 6, limit: 58893

Free Disk Space

Low free disk space watermark: 2.0 gb
Free disk space: 52.4022 gb

Totals

Connection count: 6
Queue count: 52
Virtual host count: 1

Listeners

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

ConfigMap

 data:
   operatorDefaults.conf: |
     queue_master_locator                       = min-masters
     disk_free_limit.absolute                   = 2GB
     cluster_partition_handling                 = pause_minority
     cluster_formation.peer_discovery_backend   = rabbit_peer_discovery_k8s
     cluster_formation.k8s.host                 = kubernetes.default
     cluster_formation.k8s.address_type         = hostname
     cluster_formation.target_cluster_size_hint = 3
     cluster_name                               = rabbitmq
   userDefinedConfiguration.conf: |
     total_memory_available_override_value = 1717986919
     channel_max                           = 20
     cluster_partition_handling            = pause_minority
     vm_memory_high_watermark_paging_ratio = 0.8
     disk_free_limit.relative              = 1.5
     collect_statistics_interval           = 10000
     log.console.level                     = warning
     log.console.formatter                 = json

This value seems to be off somehow...? total_memory_available_override_value = 1717986919
Operator image docker.io/bitnami/rabbitmq-cluster-operator:2.3.0-scratch-r0

RabbitMQCluster CR

   rabbitmq:
     additionalConfig: |
       channel_max = 20
       cluster_partition_handling = pause_minority
       vm_memory_high_watermark_paging_ratio = 0.8
       disk_free_limit.relative = 1.5
       collect_statistics_interval = 10000
       log.console.level = warning
       log.console.formatter = json
     additionalPlugins:
     - rabbitmq_shovel
     - rabbitmq_shovel_management
   replicas: 3
   resources:
     limits:
       cpu: 500m
       memory: 2Gi
     requests:
       cpu: 200m
       memory: 2Gi

[michaelklishin]

1717986919 * 0.4 = 687194767.6.

Per your own output the node reports

Memory high watermark setting: 0.4 of available memory, computed to: 0.6872 gb

[michaelklishin]

So the config value and the setting do match what the node logs. You have to be more specific than "off somehow". What is your evidence?

[carlosjgp]

Looks like I copied an extra digit after 1717986919 7 and when converting to GiB I was getting a value very close to the K8s node available memory.

Sorry for the false herring. It took me 3 days to notice that I copied that value incorrectly