Intermittent RabbitMQ node (Erlang VM, runtime) process termination

[ssurya]

We are facing an intermittent crash with RabbitMQ running v3.13.0 with Erlang/OTP v26.2.3 occurring once every few weeks.

It leaves a erl_crash.dump file with the following slogan:

Slogan: eheap_alloc: Cannot allocate 1573200 bytes of memory (of type "heap").

I have attached one such erl_crash.dump file with this email.
erl_crash.dump.2025-05-20.zip

This typically happens when a few hundred messages are published into the queue simultaneously on 3-4 different production systems running their own dedicated RabbitMQ instances.

The system has 15 GB of RAM available and the RAM utilization is around 20% in a normal usage, so it makes very little sense to me why the system went out-of-memory in allocating a block of just around 1.5 MB.

I have no specific steps to reproduce this as this doesn't happen on our test systems and I am able to push loads of up to 600K messages without any issues.

Can anyone help me pinpoint possible reasons of why this is happening? What RabbitMQ parameters we should tune up to avoid this from happening?

Thanks,
Shoaib.

[gomoripeti]

interesting issue indeed. I suspect memory allocated by the Erlang VM is very fragmented as the crash dump reports only ~153MB is used.
Looking at the schedulers in the crash dump only one is active executing a garbage collection for process <0.502.0> while it tries to do some logging (io_lib_pretty:printable_unicode/4 ... rabbit_logger_fmt_helpers:format_report/3)

Searching more for <0.502.0> I found

=proc:<0.457.0>
State: Waiting
Name: rabbit_memory_monitor
Message queue length: 2
...

=proc_heap:<0.457.0>
...
AA:deregister,P<0.502.0>

It looks like <0.502.0> is a queue process which terminates for some reason (deregisters from the rabbit_memory_monitor https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_amqqueue_process.erl#L355) and tries to print something big, probably its state. Pretty printing can take much more memory than the size of the data because of indentation.

But why the queue process terminates? There is something like a stacktrace here (sorry for the lengthy dump)

=proc_heap:<0.502.0>
...
2173221EEF8:t4:A1D:rabbit_classic_queue_store_v2,A16:-flush_buffer/2-fun-0-,I4,H2173221F8A8
2173221F8A8:lH2173221FFA0|H2173221FFB8
2173221FFA0:t2:A4:file,H21727DD72E8
2173221FFB8:lH21732220E10|N
21732220E10:t2:A4:line,I226 <-- line 226
2173221EF20:lH2173221F8B8|H2173221F8E0
2173221F8B8:t4:A5:lists,A5:foldl,I3,H2173221FFC8
2173221FFC8:lH21732220E28|H21732220E40
21732220E28:t2:A4:file,H2171FAEDFD0
21732220E40:lH217322223C8|N
217322223C8:t2:A4:line,I1594
2173221F8E0:lH2173221FFD8|H21732220000
2173221FFD8:t4:A1D:rabbit_classic_queue_store_v2,AC:flush_buffer,I2,H21732220E50
21732220E50:lH217322223E0|H217322223F8
217322223E0:t2:A4:file,H21727DD72E8
217322223F8:lH21732222F50|N
21732222F50:t2:A4:line,I210
21732220000:lH21732220E60|H21732220E88
21732220E60:t4:A15:rabbit_variable_queue,A7:timeout,I1,H21732222408
21732222408:lH21732222F68|H21732222F80
21732222F68:t2:A4:file,H21728864980
21732222F80:lH217322233D8|N
217322233D8:t2:A4:line,I781
21732220E88:lH21732222418|H21732222440
21732222418:t4:A15:rabbit_priority_queue,A7:timeout,I1,H21732222F90
21732222F90:lH217322233F0|H21732223408
217322233F0:t2:A4:file,H21728828280
21732223408:lH217322237E0|N
217322237E0:t2:A4:line,I388
21732222440:lH21732222FA0|H21732222FC8
21732222FA0:t4:A17:rabbit_amqqueue_process,A15:backing_queue_timeout,I1,H21732223418
21732223418:lH217322237F8|H21732223810
217322237F8:t2:A4:file,H21727DB0980
21732223810:lH21732223DB8|N
21732223DB8:t2:A4:line,I971
21732222FC8:lH21732223428|H21732223450
21732223428:t4:A17:rabbit_amqqueue_process,AB:handle_info,I2,H21732223820
21732223820:lH21732223DD0|H21732223DE8
21732223DD0:t2:A4:file,H21727DB0980
21732223DE8:lH21732224508|N
21732224508:t2:A4:line,I1778
21732223450:lH21732223830|N
21732223830:t4:AB:gen_server2,AA:handle_msg,I2,H21732223DF8
21732223DF8:lH21732224520|H21732224538
21732224520:t2:A4:file,H21728875620
21732224538:lH217322249C8|N
217322249C8:t2:A4:line,I1056
2173221E818:t2:A8:badmatch,H2173221EEE0
2173221EEE0:t2:A5:error,H2173221F890
2173221F890:t2:I0,A3:eio <-- probably the error reason

that suggests in rabbit_classic_queue_store_v2:flush_buffer/2 line 226 (https://github.com/rabbitmq/rabbitmq-server/blob/v3.13.0/deps/rabbit/src/rabbit_classic_queue_store_v2.erl#L226) a write to file fails with {badmatch, {error, eio}}. So there must be an I/O error happening for some reason.

[ssurya]

@gomoripeti Thank you so much for your response and detailed explanation.

I have tried to correlate this with two other crash dumps from earlier on from the same system but the underlying reasons seem to be different each time.

I have attached these two crash dumps here as well.
erl_crash.2024-02-25.zip
erl_crash.2024-03-05.zip

Also, more importantly, is there a way to avoid the memory allocated by the Erlang VM to become so much fragmented?

I'm using the following parameters currently.

-s "rabbit" boot -boot "start_sasl" +W w +MBas ageffcbf +MHas ageffcbf +MBlmbcs 8192 +MHlmbcs 65536 +MMscs 256 +MMmcs 30 +pc unicode +P 1048576 +t 5000000 +stbt db +zdbbl 128000 +sbwt none +sbwtdcpu none +sbwtdio none -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 -syslog logger [] -syslog syslog_error_logger false -kernel prevent_overlapping_partitions false

Should I try changing the memory allocation strategy or any other parameters? Or is there any other suggestion that could help avoid this?

Regards,
Shoaib.

[ssurya]

@michaelklishin Thank you very much for your response.

I had studied the documentation guide before posting my question here, and while there are many options and parameters that I could try and see if it helps the situation, there is nothing that I could find that will point towards the root cause of the issue we are running into.

As I had mentioned in my original post, this typically happens when a few hundred messages are published into the queue simultaneously, once every few weeks or so.

The particular system (virtual machine) that I shared the crash dumps from is running Windows Server 2019, has 15 GB of RAM available and the RAM utilization is around 20% in a normal usage and we have Nagios alerting on 80% RAM usage that it never hits. The frequency of the crashes is such that three most crashes were on 2025-05-20, 2024-03-05 and 2024-02-25.

Unfortunately, I have not been able to identify any specific steps to reproduce this problem on a test system. On the test system, I have been able to push loads of up to 600K messages without any issues and both the memory usage has stayed nominal and the no crashes occurred either.

The problem is not isolated to this system either as there are at least 4 other production systems (all running Windows Server 2019 or 2022) with their own dedicated RabbitMQ instances that have crashed over time with the same slogan.

I am slowing trying out the different options mentioned in the documentation guide, but since it takes a few weeks before the problem shows up, makes it very difficult to ascertain whether changes made have done anything positive or not.

I am in desperate need of any help or direction you or anyone else in the community could provide, because so far we have only been able to handle this is by periodically restarting the RabbitMQ service and that is obviously not a viable solution.

If there is anything further about the workload, environment or anything else that I could provide that may be helpful in explaining the scenario better, please let me know.

Thank you

Regards,
Shoaib.

[lukebakken]

this typically happens when a few hundred messages are published into the queue simultaneously

Is there anything different about these messages, especially their size?

[ssurya]

@lukebakken Thank you for your response.

Usually it's the same message that's published into the queue simultaneously for a few hundred different devices.

The same workflow gets repeated dozens of times and keeps working smoothly before all of a sudden the Erlang VM crashes because somehow it observes that the system went out-of-memory while attempting to allocate a memory chunk of 600KB or 1.5MB.

Regards,
Shoaib.

[lukebakken]

What is the average message size?

[ssurya]

It ranges from 256 bytes to around 6KB but most messages are around 1024 bytes (1KB).

[lukebakken]

@ssurya is there anything in the Windows Event Viewer that corresponds to these crashes?

[ssurya]

@lukebakken Thank you very much for pointing into that direction.

I went through the Windows Event Viewer entries on different systems that have exhibited this problem.

Searching around timestamps reported in the crash dumps, in most cases, I don't see anything relevant but at least in two cases found so far, these are the events logged:

Application popup: Windows - Out of Virtual Memory : Your system is low on virtual memory. To ensure that Windows runs properly, increase the size of your virtual memory paging file. For more information, see Help.

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: MsMpEng.exe (9008) consumed 367661056 bytes, erl.exe (11204) consumed 178958336 bytes, and messenger.exe (8696) consumed 88571904 bytes.

RabbitMQ: Restarted erlang machine.

RabbitMQ: Erlang machine seems to die continuously, not restarted.

I'm wondering if I should manually change the default size of the paging file. Can you please share your expert opinion on this as well?

Regards,
Shoaib.

[lukebakken]

I strongly suggest disabling all anti-virus to see if that resolves the issue - https://www.google.com/search?q=MsMpEng.exe+site:answers.microsoft.com

There really shouldn't be any need for that on a server, anyway, in my opinion.

Although, just after I wrote that, I googled messenger.exe - https://www.google.com/search?q=%22messenger.exe%22+what+is

Seems like messenger.exe may be a virus.

There's something up with your environment. I doubt that these Erlang crashes are due to your workload or RabbitMQ.

[ssurya]

@lukebakken Thank you for your response and for your valuable input and suggestions.

I agree with your opinion on the anti-virus and I am going to see if disabling anti-virus helps resolve the problem.

I am also going to run some tests with virtual memory configuration of the system and see if that helps reproduce the problem in the first place and then if fine-tuning it further helps in resolving the problem altogether.

As far as messenger.exe is concerned, this is the name of one of our core background NT service processes. So, it's something that I know about and is not a virus.

Regards,
Shoaib.

[lukebakken]

As far as messenger.exe is concerned, this is the name of one of our core background NT service processes. So, it's something that I know about and is not a virus.

OK, great.

Just so you know the best practice for running RabbitMQ is to not have any competing system processes, servers, or applications on the same machine.

[ssurya]

After conducting some extensive amount of testing with manually-configured low virtual memory configuration of the operating system (Windows Server), I'm able to reproduce these Erlang VM crashes quite frequently.

During investigation, it also came up as a surprise that the stock VMs from the cloud services provider that we have been using for many years had virtual memory configuration set to the "Custom" option instead of "System managed" option and it was restricted to a relatively restricted maximum size of the paging file. With the virtual memory configuration set to a the standard "System managed" option, no such issues have been reported in the past two weeks.

Besides this, I have also configured antivirus exclusions for Erlang/RabbitMQ processes and that has also helped in reducing the overall resource consumption on these servers.

While it may be a bit early to reach a final conclusion, I think I'm fairly confident that this was root cause of the problem.

Thank you so much @lukebakken, @michaelklishin and @gomoripeti for all your help and guidance in helping me narrow-down this issue.

Regards,
Shoaib.

[michaelklishin]

Thank you for reporting back. Can you name the cloud provider and the image? (it's perfectly fine if you cannot, for any reason)

[ssurya]

@michaelklishin
The cloud services provider is Rackspace and all of these four VMs were built using the default Windows Server 2019 image named "base image with updates".

I also cross-checked that this also happens to be the case with an older (about to be retired) Windows Server 2012 instance as well, but I didn't get a chance to confirm this for the Windows Server 2016 image.

Regards,
Shoaib.

[lukebakken]

@ssurya we really appreciate the follow-up!

had virtual memory configuration set to the "Custom" option instead of "System managed" option and it was restricted to a relatively restricted maximum size of the paging file.

It would have taken me a long time to think to check the virtual memory configuration, because who would ever change that??!?! Now I can add it to my list of "things to check when RabbitMQ on Windows is acting up" 👍