Preallocated memory and other memory difference between Physical vs Virtual machine

[rayate2410]

Hi,

We're running rabbitmq 3.11.13 in a containerized k8s environment. While benchmarking memory usage, we noticed difference in preallocated memory reported on management UI between physical system and virtual machine.

On physical system:

On Virtual Machine:

The usage is almost identical on these two systems. Yet we see difference between two factors:

1. Preallocated memory
2. Other memory under system.

Is there any difference in memory management in physical system vs virtual machine.

We want to predict the memory usage and accordingly set the strategy, pod limits, high watermark settings etc, and these differences between VM and physical system are kind of confusing us.

Thanks in advance.

[mkuratczyk]

This is more of an Erlang question than a RabbitMQ question. Erlang provides a lot of memory tuning options (see https://www.rabbitmq.com/runtime.html#allocators, https://www.erlang.org/doc/man/erts_alloc.html) and some of these flags might be configured differently in the Docker container for example. Some decisions can also be made by the Erlang runtime based on the resources discovered on startup.

Do not expect identical usage even if you do exactly the same things and if this a live environment with messages going in and out, even if at similar rates, these values will be different to some extent. In both cases your usage is very low, so I guess the key question is whether it ever goes significantly above these levels.

[rayate2410]

Although i have not seen it going above 48-51MB, the differences in memory usage under different systems (under identical load) is a bit difficult for me to understand.

The question is are there any best practices documented to run inside docker container?
Also, I'd like to understand on what bases erlang decides to pre-allocate X amount of memory, this will help me understand and predict the memory requirements for the container in the worst case scenarios.

[mkuratczyk]

There is no simple answer this. Memory management is a very complex topic - there are operating-system level aspects, virtualization aspects, containerization aspects, (Erlang, in this case) runtime aspects and then there is actual application (RabbitMQ) usage, but since Erlang is a garbage-collected language, even here the usage is not always obvious, since it's not explicit. On top of this, all these layers have many configuration options and change from version to version.

If you are interested in the details, well, brace yourself for a long journey. I certainly don't claim to understand all aspects of the memory management involved in running RabbitMQ. :)
You can start here for example: https://blog.stenmans.org/theBeamBook/#CH-Memory

Unless this is some kind of embedded system, I'd say a 1GB is a minimum I'd go with anyway, so if that's many times above your actual observed usage, you should be fine. But it's up to you to test under real/representative workload. Moreover, since things change from version to version, this is not something you can do once and forget about (eg. in 3.12 we completely changed how memory is utilized by classic queues, although in general this should lead to lower usage).

[michaelklishin]

There are also Erlang/BEAM conference talks on the topic, like this one.