erl crash dump analysis request

[albu70]

I see that RabbitMQ server v3.11.10 crashes after some time when sitting idle. There is a dump file that gets created at that time which I have attached as a log file.

Any pointers as to why this might be happening would be greatly appreciated.
erl_crash.log

Thanks.

[michaelklishin]

According to this crash dump file (they are rarely useful, node logs are, but none were provided),
the node terminated very early on. It is not necessarily a RabbitMQ node, could be a CLI process.

It does have RabbitMQ modules on the code path but it barely has any processes running, and none of them are RabbitMQ processes:

It has managed to run code loader and start a logger. That's it.

[albu70]

[email protected]

[lukebakken]

Your log file shows enomem when trying to read data from disk:

2023-05-02 12:59:20.683000-07:00 [notice] <0.227.0> Logging: configured log handlers are now ACTIVE
2023-05-02 12:59:23.908000-07:00 [info] <0.227.0> ra: starting system quorum_queues
2023-05-02 12:59:23.909000-07:00 [info] <0.227.0> starting Ra system: quorum_queues in directory: c:/Users/Admin/AppData/Roaming/RabbitMQ/db/rabbit@myPC-2-mnesia/quorum/rabbit@myPC-2
2023-05-02 12:59:23.910000-07:00 [info] <0.295.0> ra system 'quorum_queues' running pre init for 0 registered servers
2023-05-02 12:59:23.911000-07:00 [info] <0.298.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2023-05-02 12:59:23.911000-07:00 [notice] <0.310.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2023-05-02 12:59:23.912000-07:00 [error] <0.310.0>   crasher:
2023-05-02 12:59:23.912000-07:00 [error] <0.310.0>     initial call: ra_log_wal:init/1
2023-05-02 12:59:23.912000-07:00 [error] <0.310.0>     pid: <0.310.0>
2023-05-02 12:59:23.912000-07:00 [error] <0.310.0>     registered_name: []
2023-05-02 12:59:23.912000-07:00 [error] <0.310.0>     exception exit: {could_not_read_wal_chunk,enomem}
2023-05-02 12:59:23.912000-07:00 [error] <0.310.0>       in function  gen_batch_server:init_it/6 (src/gen_batch_server.erl, line 182)

This means that the Erlang VM could not allocate memory to read the data, which is a very strange error. Your Windows environment must not have much free memory.

Note that the latest boot at 2023-05-23 19:39:21.901000-07:00 appears to have been successful.

cc @kjnilsson because you might find this interesting.

[albu70]

Hmm, that's interesting. I have a 16 GB system where almost 40-50% memory is free.

How much memory does erlang need to host rabbitmq? Is there a way to configure it?

Thanks.

[michaelklishin]

How much memory your nodes need depends on the workload, what exactly your clients do, and changes over time.

While looking at the Erlang codebase, I see that enomem does not necessarily mean that there is no more physical memory left on the machine, or even available to the runtime. In a couple of cases
it means that an allocator failed and somehow this was not retried (not that it should or can be retried in every case).

Having 40-50% of memory free does not mean that it was that way at the time of the exception.
It also does not mean that the runtime can use it. There can be restrictions enforced by various
Windows tools, SELinux, and so on.

This blog post mentions what kind of configurations are used for our load tests. It can be an overkill for your needs but you can take a look as a reference point. The nodes in these tests are pushed close to their limits on one metric or another.