Node memory footprint on CentOS Stream 9

[michaelklishin]

Team RabbitMQ and community members have recently identified a curious scenario where a freshly started node could consume a surprisingly high amount of memory, say, 1.5 GiB or so. We'd like to share our findings with the community and explain what short term and longer term workarounds are available.

Some latest RPM-based distributions, such as RHEL 9 and CentOS Stream 9, ship a recent version of systemd that sets the default open file handle limit is set to 1073741816 or about one billion. This is much higher than the default used by older distributions such as CentOS 8, RHEL 8.x, and even modern RPM-based distributions such as Rocky Linux or modern Fedora.

For a lot of software this doesn't change anything. However, the Erlang runtime will allocate more memory upfront on systems with a very high limit. This leads to surprisingly high footprint of newly started RabbitMQ nodes without any data or meaningful client activity.

There are two ways to mitigate this problem:

• Lower the file handle limit
• Set the ERL_MAX_PORTS environment variable to a lower value

What value would be more appropriate for your given environment depends on the workload. Default values in the 50,000 to 100,000 range should support plenty of concurrent client connections, queues and streams for many cases without causing excessive upfront memory allocation by the runtime.

[gomoripeti]

This is great insight, thanks for all involved.

I have two minor questions.
Though ERL_MAX_PORTS works I cannot find it documented anywhere (looks like it was deprecated in R16B) instead there is the extra erl argument +Q. Could +Q be mentioned in the blog as well? Did you choose to use the env var because it is easier to override?
Since OTP 21 the file driver does not use erlang ports, and with the introduction of the socket NIF soon network connections won't either. (So it almost feels like an OTP bug to me that the port table size is initialized based on FD limit, as it can be a great overestimation) Could ERL_MAX_PORTS be set to an even more conservative value based on just the expected network connections (and ignoring the number of file handles that RabbitMQ uses)? Apart from the number of external connections (clients, management UI etc) what else uses sockets internally? (I can think of inter-node erlang connections, a stream's replication connection)

[lukebakken]

(looks like it was deprecated in R16B)

...and never removed, hmm -

https://github.com/erlang/otp/blob/master/erts/doc/src/notes.xml#L18251-L18254

Did you choose to use the env var because it is easier to override?

It's just out of familiarity. Seems like +Q is preferable. https://www.erlang.org/doc/man/erl.html

We just merged this PR #5684 so it looks like I should revisit how we address this issue.

I'll consider opening an issue in https://github.com/erlang/otp.git with regard to how the port table is initialized.