The RabbitMQ management port fails to listen

[dormanze]

Environments:

rabbitmq version: 3.11.9
erlang version: 25.22

Symptoms:

In my environment, two RabbitMQ containers and nodes have been powered off. After powering up, I found that the management port of one of the instances failed to listen.Node details of Rabbitmqservice-0 are empty.

Possible Causes:

I think the data synchronization failure is caused by abnormal power-off, because I see many abnormal logs during power-off.

2023-08-18 10:08:19.047 [error] <0.3006.0> ** Generic server <0.3006.0> terminating
** Last message in was timeout
** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"jobs.frontend-gateway">>},true,false,none,[],<0.3006.0>,[<17210.23131.129>],[],['rabbit@rabbitmqservice-1'],[{vhost,<<"/">>},{name,<<"ha-allqueue">>},{pattern,<<"^">>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"all">>},{<<"ha-promote-on-failure">>,<<"always">>},{<<"ha-promote-on-shutdown">>,<<"always">>},{<<"ha-sync-mode">>,<<"automatic">>}]},{priority,0}],undefined,[{<17210.23135.129>,<17210.23131.129>},{<0.3012.0>,<0.3006.0>}],[],live,0,[],<<"/">>,#{user => <<"rmq-internal">>},rabbit_classic_queue,#{}},none,false,rabbit_mirror_queue_master,{state,{resource,<<"/">>,queue,<<"jobs.frontend-gateway">>},<0.3012.0>,<0.23501.5>,rabbit_priority_queue,{passthrough,rabbit_variable_queue,{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{7592,{[{msg_status,7591,<<84,107,150,159,215,212,116,250,62,211,100,93,186,183,217,246>>,{basic_message,{resource,<<"/">>,exchange,<<"exchange.scheduler.jobs">>},[<<"rk.jobs.frontend-gateway">>],{content,60,{'P_basic',<<"application/json">>,<<"UTF-8">>,[{<<"b3">>,longstr,<<"715fbae945e545db-bbe71828254cf871-0">>},{<<"__TypeId__">>,longstr,<<"com.huawei.telcloud.scheduler.message.JobExecutionMessage">>}],2,0,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},<<248,0,16,97,112,112,108,105,99,97,116,105,111,110,47,106,115,111,110,5,85,84,70,45,56,0,0,0,116,2,98,51,83,0,0,0,35,55,49,53,102,98,97,101,57,52,53,101,53,52,53,100,98,45,98,98,101,55,49,56,50,56,50,53,52,99,102,56,55,49,45,48,10,95,95,84,121,112,101,73,100,95,95,83,0,0,0,57,99,111,109,46,104,117,97,119,101,105,46,116,101,108,99,108,111,117,100,46,115,99,104,101,100,117,108,101,114,46,109,101,115,115,97,103,101,46,74,111,98,69,120,101,99,117,116,105,111,110,77,101,115,115,97,103,101,2,0>>,rabbit_framing_amqp_0_9_1,[<<"{\"serviceName\":\"frontend-gateway\",\"jobName\":\"recordExpireSessionLog\",\"action...">>]},...},...},...],...}},...}},...},...}
** Reason for termination ==
** {{badmatch,{error,enospc}},[{rabbit_queue_index,sync,1,[{file,"src/rabbit_queue_index.erl"},{line,442}]},{rabbit_variable_queue,timeout,1,[{file,"src/rabbit_variable_queue.erl"},{line,904}]},{rabbit_priority_queue,timeout,1,[{file,"src/rabbit_priority_queue.erl"},{line,388}]},{rabbit_mirror_queue_master,timeout,1,[{file,"src/rabbit_mirror_queue_master.erl"},{line,408}]},{rabbit_amqqueue_process,backing_queue_timeout,1,[{file,"src/rabbit_amqqueue_process.erl"},{line,964}]},{rabbit_amqqueue_process,handle_info,2,[{file,"src/rabbit_amqqueue_process.erl"},{line,1717}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}
2023-08-18 10:08:19.083 [info] <0.2143.17> rabbit_exchange_type_topic:trie_binding_op:operation type is : delete , exchange name is : {resource,<<"/">>,exchange,<<"mateinfo.temp.topic">>} , routing key is : <<"system.plane-channel-msg-adc-runtime">> , node id is : <<255,216,245,230,57,94,94,51,203,139,42,170,26,81,253,60>> .
2023-08-18 10:08:19.087 [error] <0.2190.17> Restarting crashed queue 'jobs.frontend-gateway' in vhost '/'.
2023-08-18 10:08:19.104 [error] <0.2371.0> ** Generic server mnesia_sync terminating

Possible Solutions:

Restart RabbitMQ.
However, I think rabbitmq should be self-healing in this abnormal scenario.

[mkuratczyk]

enospc - you've run out of disk space

[dormanze]

Indeed, I see that the disk monitoring is over 100% instantaneously. Thank you for your answer.