RabbitMQ becoming unresponsive and unable to recover

[adriano-oliveira-wr]

We have an issue in the RabbitMQ version 3.6.5 Erlang 18.3.4.4 in which the service becomes unresponsive.
There is a service monitor that uses the "rabbitmqctl status" to be a heartbeat and at some point it fails and then it calls a recovery procedure: it calls the "rabbitmqctl stop", wait to check if the amqp port is released, then call start the server.
In this process, the stop path fails because the port is still hold, however, the server is not running (at least not responding properly the status), then it is not able to start right away, after some retries eventually it starts, but just after a few seconds it dies again and the recovery cycle repeats (until eventually it logs a crash dump in the logs).
Bottom line is that RabbitMQ enters a bad state in which it stays kind of messy and does not recover within the required window.
Do you know what could be causing that or, maybe, point to some issue that has been resolved in newer versions that could be fixing similar error?

We got these error logs from RabbitMQ:

startup_err:

Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{k

Crash dump is being written to: erl_crash.dump...done

init terminating in do_boot ()
/usr/lib/rabbitmq/bin/rabbitmq-server: line 237: 415264 User defined signal 2   start_rabbitmq_server "$@"

startup_log:

{error_logger,{{},{}},"Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",eaddrinuse]}

{error_logger,{{},{}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}},{type,permanent}]}

[email protected]

=SUPERVISOR REPORT====  ===
     Supervisor: {local,rabbit_epmd_monitor_sup}
     Context:    start_error
     Reason:     {{badmatch,noport},
                  [{rabbit_epmd_monitor,init,1,
                       [{file,"src/rabbit_epmd_monitor.erl"},{line,56}]},
                   {gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},
                   {proc_lib,init_p_do_apply,3,
                       [{file,"proc_lib.erl"},{line,240}]}]}
     Offender:   [{pid,undefined},
                  {name,rabbit_epmd_monitor},
                  {mfargs,{rabbit_epmd_monitor,start_link,[]}},
                  {restart_type,transient},
                  {shutdown,30000},
                  {child_type,worker}]


=CRASH REPORT====  ===
  crasher:
    initial call: application_master:init/4
    pid: <0.130.0>
    registered_name: []
    exception exit: {bad_return,
                     {{rabbit,start,[normal,[]]},
                      {'EXIT',
                       {error,
                        {{shutdown,
                          {failed_to_start_child,rabbit_epmd_monitor,
                           {{badmatch,noport},
                            [{rabbit_epmd_monitor,init,1,
                              [{file,"src/rabbit_epmd_monitor.erl"},
                               {line,56}]},
                             {gen_server,init_it,6,
                              [{file,"gen_server.erl"},{line,328}]},
                             {proc_lib,init_p_do_apply,3,
                              [{file,"proc_lib.erl"},{line,240}]}]}}},
                         {child,undefined,rabbit_epmd_monitor_sup,
                          {rabbit_restartable_sup,start_link,
                           [rabbit_epmd_monitor_sup,
                            {rabbit_epmd_monitor,start_link,[]},
                            false]},
                          transient,infinity,supervisor,
                          [rabbit_restartable_sup]}}}}}}
      in function  application_master:init/4 (application_master.erl, line 134)
    ancestors: [<0.129.0>]
    messages: [{'EXIT',<0.131.0>,normal}]
    links: [<0.129.0>,<0.7.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 197
  neighbours:

[lukebakken]

Thanks for using RabbitMQ. Unfortunately, we do not have time to provide support for an out-of-date version of RabbitMQ. Please upgrade - https://www.rabbitmq.com/versions.html