Memory footprint in environments where clients repeatedly run into channel exceptions

[DenZen1988]

Describe the bug

If the vhost(s) of clients do not have set the correct exchange:

no exchange 'ProxyExchange' in vhost 'cache'

and there are active connection tries (e.g. faulty configuration) it looks like there are some memory leftovers produced.
Those seem to be not seen by the rabbitmq itself anymore.

Loglines (every 5 seconds form a service connecting to the rabbitmq):

Mar 8 00:00:22 <hostname> <servicename>[4317]: E: Failed to bind to exchange ProxyExchange: (404, "NOT_FOUND - no exchange 'ProxyExchange' in vhost 'cache'")
Mar 8 00:00:27 <hostname> <servicename>[4317]: I: Creating outbound channel
Mar 8 00:00:27 <hostname> <servicename>[4317]: I: Creating inbound channel
Mar 8 00:00:27 <hostname> <servicename>[4317]: I: Binding to exchange ProxyExchange, routing_key DesiredState.Proxy.XX, queue amq.gen-GXaTGMIodoDjG35W96Q23A for broadcasts
Mar 8 00:00:27 <hostname> <servicename>[4317]: W: Received remote Channel.Close (404): "NOT_FOUND - no exchange 'ProxyExchange' in vhost 'cache'" on <Channel number=2 OPEN conn=<SelectConnection OPEN socket=('XXX.XXX.XXX.XXX', 13472)->('XXX.XXX.XXX.XXX', 5671) params=<ConnectionParameters host=<hostname> port=5671 virtual_host=cache ssl=True>>>
Mar 8 00:00:27 <hostname> <servicename>[4317]: E: Failed to bind to exchange ProxyExchange: (404, "NOT_FOUND - no exchange 'ProxyExchange' in vhost 'cache'")

Loglines from the rabbitmq host:

2023-03-06 12:41:37.366823+00:00 [info] <0.11152.180> connection <0.11152.180> (XXX.XXX.XXX.XXX:45482 -> XXX.XXX.XXX.XXX:5671): user '<username>' authenticated and granted access to vhost 'cache'
2023-03-06 12:41:37.387989+00:00 [error] <0.11222.180> Channel error on connection <0.11152.180> (XXX.XXX.XXX.XXX:45482 -> XXX.XXX.XXX.XXX:5671, vhost: 'cache', user: '<username>'), channel 2:
2023-03-06 12:41:37.387989+00:00 [error] <0.11222.180> operation queue.bind caused a channel exception not_found: no exchange 'ProxyExchange' in vhost 'cache'

The error appeared roughly around every 6 hours by 8 hosts connecting to 1 rabbitmq instance/server.
After roughly 6 hours the memory exhausted (machine OOM killer) and rabbitmq did not report that it used that much memory.

Reproduction steps

1. Run 1 instance of rabbitmq
2. Configure 8 clients to connect to the rabbitmq to a non existing vhost
3. Wait for roughly 6 hours
4. The OOM killer should set in
5. As soon a the clients are removed or configured correctly the issue disappears

Expected behavior

The expected behaviour should either be that rabbitmq reports these memory leftovers so an admin can take action or those should be included in a garbage collector and be cleaned up automatically.

Additional context

No response

[lukebakken]

What version of RabbitMQ and Erlang are you using?

[DenZen1988]

When the issue appeared the versions have been:

• Erlang: 1:25.2.1-1
• RabbitMQ: 3.11.7-1

[michaelklishin]

Reasoning About Memory Use explains how to produce a breakdown of what uses the memory.

According to the log above, your code

1. Opens a connection
2. Opens a channel
3. Runs into a channel-level exception on it (that closes the channel but DOES NOT close the connection)

and then… according to the log, the connection is leaked.

Connection churn is another resource-intensive scenario that has nothing to do with "RabbitMQ garbage collection" but
rather with how the kernel manages TCP connections, in particular in case of connection churn.

See Connection Lifecycle Events for another source of information about what is going on in your system.

Clients can leak resources. In some cases, RabbitMQ cannot do anything about that (an open connection is an open connection), and cannot be blamed for the resource leak.

[michaelklishin]

@denis-walther without an executable way to reproduce (even in a few hours) all we can suggest is using the doc guides above to narrow down if this may be as trivial as a connection leak, or a case of high connection churn.

Please put together an example repo we can run.

[michaelklishin]

Specifically this part

As soon a the clients are removed or configured correctly the issue disappears

makes me think that it is a good old connection leak by your own apps that you blame on RabbitMQ's "garbage collector".

FTR, you can trigger a system-wide GC run (for all Erlang processes on the node) using CLI tools:

rabbitmqctl force_gc

If that does not help even after a brief period of time, the issue is not with garbage collection
but is with either resource leaks performed by applications, or memory fragmentation in the runtime, a very complex topic with many settings that can be tuned but no one-size-fits-all solution for every workload.