Worse RabbitMQ performance when sending less data and performing fewer tasks

[ArthurKerst]

I have RabbitMQ running in a Windows Docker container with Erlang (version 25.3) and RabbitMQ (version 3.11.18). This container operates in process isolation with four enabled schedulers. Mainly topic exchanges are used. In my application, composed of services running in Docker containers, there are two modes: recording and non-recording. During recording, the system load is higher, and more data is transmitted through RabbitMQ. However, in non-recording mode, where the application performs fewer tasks and less data is send through RabbitMQ, RabbitMQ performance degrades. I confirmed this by running a monitoring process that sends a message with a timestamp, receives the message, and calculates the time difference between sending and receiving.

In recording mode, the time difference is approximately 10 ms, but in non-recording mode, the time differences are around 2000 ms at times (see graph 3 in the added image). I experimented with various settings for the Erlang runtime system, and although increasing the number of schedulers from 2 to 4 improved performance to some extent, the issue persists. What could be causing this behavior, and how can I enhance the performance?

[michaelklishin]

It's not clear to me what exactly your application does and how do you measure "performance". If an application voluntarily publishes less, some metrics will be lower.

When the node is not always under load, the runtime may hibernate processes, which introduces a "waking up" cost. There are settings that can control for how long a node will keep its schedules busy spinning to avoid this. This obviously will be counterproductive on mostly idle systems, which is why we recommend shortening or disabling the idling period for mostly idle nodes.

This "unhibernation" cost will mostly be observable as a latency increase, and won't be permanent. Its effect on throughput will be less visible on median/mean throughput metrics.

Note that two cores is not an adequate number of cores for production systems.

[lukebakken]

@ArthurKerst - it would be great to know more about the environment you're using. Your statement that you're using a "Windows Docker container" implies that RabbitMQ is running in a container on Windows. Is that correct?

• Which version of Windows is the host system?
• Which docker container are you using for RabbitMQ? The official one? https://hub.docker.com/_/rabbitmq
• Which backend are you using to run the containers? Hyper-V or WSL2 (https://docs.docker.com/desktop/wsl/)

My guess, based on the limited information given, is that the virtualization environment reduces the resources available to RabbitMQ when the workload is lower to conserve those resources. There may also be a "noisy neighbor" situation going on here.