Node is running but cannot accept new connections on Windows due to an eacces error

[jabbera]

Describe the bug

Preface: Thanks for reading

We are unsure what is causing this issue, but we are getting in this situation often and need some assistance. We have a single node deployment. Symptoms are first reported as inability to connect to rabbitmq. Upon looking in the logfile we see erlang appears to have crashed and a dump has been written:

Error

2024-08-01 16:58:39.666000-04:00 [error] <0.32173095.0> Error in process <0.32173095.0> on node rabbit@MARPRDFAB611 with exit value:
2024-08-01 16:58:39.666000-04:00 [error] <0.32173095.0> {{badmatch,{error,eacces}},
2024-08-01 16:58:39.666000-04:00 [error] <0.32173095.0>  [{mnesia_log,save_decision_tab,1,[{file,"mnesia_log.erl"},{line,537}]},
2024-08-01 16:58:39.666000-04:00 [error] <0.32173095.0>   {mnesia_dumper,perform_dump,2,[{file,"mnesia_dumper.erl"},{line,151}]},
2024-08-01 16:58:39.666000-04:00 [error] <0.32173095.0>   {mnesia_controller,dump_and_reply,2,
2024-08-01 16:58:39.666000-04:00 [error] <0.32173095.0>                      [{file,"mnesia_controller.erl"},{line,2147}]}]}
2024-08-01 16:58:39.666000-04:00 [error] <0.32173095.0> 
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>     supervisor: {local,mnesia_kernel_sup}
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>     errorContext: child_terminated
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>     reason: killed
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>     offender: [{pid,<0.304.0>},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {id,mnesia_locker},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {mfargs,{mnesia_locker,start,[]}},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {restart_type,permanent},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {significant,false},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {shutdown,3000},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {child_type,worker}]
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0> 
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>     supervisor: {local,mnesia_kernel_sup}
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>     errorContext: shutdown
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>     reason: reached_max_restart_intensity
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>     offender: [{pid,<0.304.0>},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {id,mnesia_locker},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {mfargs,{mnesia_locker,start,[]}},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {restart_type,permanent},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {significant,false},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {shutdown,3000},
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0>                {child_type,worker}]
2024-08-01 16:58:43.791000-04:00 [error] <0.301.0> 
2024-08-01 16:58:43.791000-04:00 [error] <0.299.0> Mnesia(rabbit@MARPRDFAB611): ** ERROR ** (core dumped to file: "d:/RabbitMQ/data/MnesiaCore.rabbit@MARPRDFAB611_1722_545923_795898")
<SNIP>

after this we see a repeated error:

2024-08-01 17:10:54.604000-04:00 [info] <0.32209426.0> accepting AMQP connection <0.32209426.0> (10.20.34.174:51430 -> 10.20.32.199:5672)
2024-08-01 17:10:54.604000-04:00 [warning] <0.32209426.0> Mnesia->Khepri fallback handling: Mnesia function failed 99 times. Possibly an infinite retry loop; trying one last time
2024-08-01 17:10:54.604000-04:00 [error] <0.32209426.0>   crasher:
<SNIP>

until we restart the rabbitmq process.

Configuration is as follows:

2024-07-29 13:29:46.132000-04:00 [info] <0.249.0> Khepri leader elected
2024-07-29 13:29:46.508000-04:00 [notice] <0.289.0> RabbitMQ metadata store: candidate -> leader in term: 8 machine version: 0
2024-07-29 13:29:47.119000-04:00 [info] <0.249.0> 
2024-07-29 13:29:47.119000-04:00 [info] <0.249.0>  Starting RabbitMQ 3.13.2 on Erlang 26.2.5 [jit]
2024-07-29 13:29:47.119000-04:00 [info] <0.249.0>  Copyright (c) 2007-2024 Broadcom Inc and/or its subsidiaries
2024-07-29 13:29:47.119000-04:00 [info] <0.249.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
2024-07-29 13:29:47.135000-04:00 [info] <0.249.0> 
2024-07-29 13:29:47.135000-04:00 [info] <0.249.0>  node           : rabbit@MARPRDFAB611
2024-07-29 13:29:47.135000-04:00 [info] <0.249.0>  home dir       : c:/Windows/system32/config/systemprofile
2024-07-29 13:29:47.135000-04:00 [info] <0.249.0>  config file(s) : d:/RabbitMQ/data/config/advanced.config
2024-07-29 13:29:47.135000-04:00 [info] <0.249.0>                 : d:/RabbitMQ/data/config/rabbitmq.conf
2024-07-29 13:29:47.135000-04:00 [info] <0.249.0>  cookie hash    : 68SvG0BAENDv/w+jeUhpsA==
2024-07-29 13:29:47.135000-04:00 [info] <0.249.0>  log(s)         : <stdout>
2024-07-29 13:29:47.135000-04:00 [info] <0.249.0>                 : d:/RabbitMQ/data/logging/[email protected]
2024-07-29 13:29:47.135000-04:00 [info] <0.249.0>  data dir       : d:/RabbitMQ/data/mnesia/rabbit@MARPRDFAB611-mnesia
2024-07-29 13:29:48.245000-04:00 [info] <0.249.0> Running boot step pre_boot defined by app rabbit
2024-07-29 13:29:48.245000-04:00 [info] <0.249.0> Running boot step rabbit_global_counters defined by app rabbit
2024-07-29 13:29:48.245000-04:00 [info] <0.249.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2024-07-29 13:29:48.245000-04:00 [info] <0.249.0> Running boot step rabbit_core_metrics defined by app rabbit

Happy to provide any additional information as requested.

Thanks again!

[lukebakken]

Hello, thanks for using RabbitMQ.

eacces is a permission error. I see that you're on Windows, so the most likely culprit is some sort of security or anti-virus software that interferes with RabbitMQ.

Disable ANY of these sorts of programs on your RabbitMQ servers, and see if the issue continues.

Just FYI, any time you report an issue with software, it's always good to provide information about how long the system worked without issue, or if the issue was always present.

[jabbera]

Thank you so much for getting back to me. I really appreciate it. We are currently removing all our AV products once I get through the red tape.