Kernel page cache footprint keeps growing

[Kraego]

Setup

• image tag: 3.12.9-management-alpine
• 0 Queues, just one durable exchange
• Environment: Openshift Cluster
• Configurationfiles (k8 configmap):

  apiVersion: v1
  
  kind: ConfigMap
  metadata:
    name: rabbitmq-config
  data:
    enabled_plugins: |-
      [rabbitmq_management,rabbitmq_prometheus,rabbitmq_tracing].
    rabbitmq.conf: |-
      listeners.tcp.default = 5672
      log.file.level = info
      vm_memory_high_watermark.absolute = 256MiB
  

Memory Consumption

The strange thing is that there is no Queue nor is there any traffic, but the memory consumption rises until the pod gets killed by the OOM.

• Queues:

/ $ rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
/ $ 

• Exchanges:

/ $ rabbitmqctl list_exchanges
Listing exchanges for vhost / ...
name    type
amq.rabbitmq.trace      topic
Registration    fanout
        direct
amq.match       headers
amq.topic       topic
amq.headers     headers
amq.fanout      fanout
amq.direct      direct

• Memory usage: The cluster metric - container_memory_working_set_bytes at this time was 306MB

/ $ rabbitmq-diagnostics memory_breakdown
Reporting memory breakdown on node rmq-host@rmq-host...
reserved_unallocated: 0.0779 gb (51.62%)
code: 0.0389 gb (25.79%)
other_system: 0.0222 gb (14.73%)
other_proc: 0.0152 gb (10.09%)
metrics: 0.0017 gb (1.15%)
other_ets: 0.0016 gb (1.06%)
atom: 0.0015 gb (0.98%)
plugins: 0.001 gb (0.68%)
binary: 0.0005 gb (0.3%)
msg_index: 0.0004 gb (0.24%)
mgmt_db: 0.0003 gb (0.21%)
mnesia: 0.0001 gb (0.05%)
connection_other: 0.0 gb (0.03%)
quorum_ets: 0.0 gb (0.02%)
quorum_queue_procs: 0.0 gb (0.0%)
quorum_queue_dlx_procs: 0.0 gb (0.0%)
stream_queue_procs: 0.0 gb (0.0%)
stream_queue_replica_reader_procs: 0.0 gb (0.0%)
connection_readers: 0.0 gb (0.0%)
connection_writers: 0.0 gb (0.0%)
connection_channels: 0.0 gb (0.0%)
queue_procs: 0.0 gb (0.0%)
queue_slave_procs: 0.0 gb (0.0%)
stream_queue_coordinator_procs: 0.0 gb (0.0%)
allocated_unused: 0.0 gb (0.0%)

• For completeness - whole status:

Status of node rmq-host@rmq-host ...
Runtime

OS PID: 13
OS: Linux
Uptime (seconds): 68724
Is under maintenance?: false
RabbitMQ version: 3.12.9
RabbitMQ release series support status: supported
Node name: rmq-host@rmq-host
Erlang configuration: Erlang/OTP 25 [erts-13.2.2.4] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit]
Crypto library: OpenSSL 3.1.4 24 Oct 2023
Erlang processes: 446 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_prometheus
 * accept
 * prometheus
 * rabbitmq_tracing
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
 * amqp_client
 * cowboy
 * cowlib

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rmq-host@rmq-host
Raft data directory: /var/lib/rabbitmq/mnesia/rmq-host@rmq-host/quorum/rmq-host@rmq-host

Config files

 * /etc/rabbitmq/rabbitmq.conf
 * /etc/rabbitmq/conf.d/10-defaults.conf

Log file(s)

 * /var/log/rabbitmq/[email protected]
 * <stdout>

Alarms

(none)

Memory

Total memory used: 0.1512 gb
Calculation strategy: rss
Memory high watermark setting: 0.2684 gb, computed to: 0.2684 gb

reserved_unallocated: 0.0774 gb (51.23 %)
code: 0.0389 gb (25.76 %)
other_system: 0.0222 gb (14.71 %)
other_proc: 0.0153 gb (10.1 %)
metrics: 0.0017 gb (1.15 %)
other_ets: 0.0016 gb (1.06 %)
atom: 0.0015 gb (0.98 %)
plugins: 0.001 gb (0.66 %)
binary: 0.0005 gb (0.31 %)
msg_index: 0.0004 gb (0.24 %)
mgmt_db: 0.0003 gb (0.21 %)
mnesia: 0.0001 gb (0.05 %)
connection_other: 0.0 gb (0.03 %)
quorum_ets: 0.0 gb (0.02 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
allocated_unused: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 2, limit: 1048479
Sockets: 0, limit: 943629

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 2.1475 gb

Totals

Connection count: 0
Queue count: 0
Virtual host count: 1

Listeners

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: 0.0.0.0, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Research Done

• Calling rabbitmqctl force_gc releases approximately 6MB
• There was also a thread (https://groups.google.com/g/rabbitmq-users/c/v630G6OCxuU) discussion a memory leak (regarding lazy queues, limiting the index, paging, ...) since we don't have any queues imho I think that's irrelevant.
• There was a possible Leak in Erlang: https://groups.google.com/g/rabbitmq-users/c/UE-wxXerJl8, this was fixed in OTP 24.2.1 (https://github.com/erlang/otp/releases/tag/OTP-24.2.1) the used image uses 25.3.2.1
• What I've recognized so far is that rabbitMq had problems detecting actual available memory in the pod: Memory high watermark setting: 0.4 of available memory, computed to: 54.0295 gb (real available is 1GiB, so I configured an absolute watermark). Maybe erlang has a similar Problem and that's the reason why the GC don't kicks in - but that's just a wild assumption since my erlang knowledge is zero.

[michaelklishin]

@Kraego this is not a memory leak. This is almost certainly due to the fact that some Kubernetes versions count kernel page cache as pod' memory footprint, which is incorrect because the page cache is managed by the kernel and no the application.

Any environment that use streams will observe more and more memory used by kernel page cache over time,
and the kernel clears it when it sees fit.

There is #7362 and at least a couple more discussions that go into a lot of detail about how to inspect effective kernel page cache.

[Kraego]

Thank you for the clarification.

[Kraego]

We are currently on Kubernetes v1.24 ... this problem should be fixed with a linux distro with cgroupv2 and a upate to kubernetes 1.25, ... see kubernetes/kubernetes#43916 (comment) and https://kubernetes.io/blog/2022/08/31/cgroupv2-ga-1-25/#:~:text=cgroup%20v2%20provides%20a%20unified,has%20graduated%20to%20general%20availability