rabbitmq3.10.2 crash,help!!!

[kmath163]

rabbitmqctl status

`rabbitmqctl status` output

root@scrm-rabbit:/var/log/rabbitmq# rabbitmqctl status
Status of node rabbit@scrm-rabbit ...
Stack trace: 

** (FunctionClauseError) no function clause matching in RabbitMQ.CLI.Core.Memory.formatted_watermark/1
    (rabbitmqctl 3.8.0-dev) lib/rabbitmq/cli/core/memory.ex:56: RabbitMQ.CLI.Core.Memory.formatted_watermark(nil)
    (rabbitmqctl 3.8.0-dev) lib/rabbitmq/cli/ctl/commands/status_command.ex:228: RabbitMQ.CLI.Ctl.Commands.StatusCommand.result_map/1
    (rabbitmqctl 3.8.0-dev) lib/rabbitmq/cli/ctl/commands/status_command.ex:68: RabbitMQ.CLI.Ctl.Commands.StatusCommand.output/2
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:153: RabbitMQCtl.maybe_run_command/3
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:121: anonymous fn/5 in RabbitMQCtl.do_exec_parsed_command/5
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:584: RabbitMQCtl.maybe_with_distribution/3
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:93: RabbitMQCtl.exec_command/2
    (rabbitmqctl 3.8.0-dev) lib/rabbitmqctl.ex:31: RabbitMQCtl.main/1

Error:
:function_clause

rabbitmq error log print:

2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0> ** Generic server vm_memory_monitor terminating
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0> ** Last message in was update
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0> ** When Server state == {state,67454459904,26981783961,169623552,0.4,10
00,
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                                #Ref<0.1108655695.3744202754.99866>,fals
e,
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                                {#Fun<rabbit_alarm.0.73210180>,
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                                 #Fun<rabbit_alarm.1.73210180>},
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                                {unix,linux},
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                                "19",4096,
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                                [47,112,114,111,99,47,"19",47,115,116,97
,116,
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                                 109]}
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0> ** Reason for termination ==
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0> ** {{badmatch,{error,enfile}},
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>     [{vm_memory_monitor,read_proc_file,1,
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                         [{file,"vm_memory_monitor.erl"},{line,580}]},
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>      {vm_memory_monitor,get_process_memory_using_strategy,2,
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                         [{file,"vm_memory_monitor.erl"},{line,283}]},
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>      {vm_memory_monitor,update_process_memory,1,
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                         [{file,"vm_memory_monitor.erl"},{line,266}]},
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>      {vm_memory_monitor,internal_update,1,
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                         [{file,"vm_memory_monitor.erl"},{line,409}]},
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>      {vm_memory_monitor,handle_info,2,
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>                         [{file,"vm_memory_monitor.erl"},{line,230}]},
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>      {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>      {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0>      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2023-08-10 05:36:16.468978+00:00 [error] <0.2181.0> 
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>   crasher:
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     initial call: vm_memory_monitor:init/1
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     pid: <0.2181.0>
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     registered_name: vm_memory_monitor
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     exception error: no match of right hand side value {error,enfile}
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>       in function  vm_memory_monitor:read_proc_file/1 (vm_memory_monito
r.erl, line 580)
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>       in call from vm_memory_monitor:get_process_memory_using_strategy/
2 (vm_memory_monitor.erl, line 283)
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>       in call from vm_memory_monitor:update_process_memory/1 (vm_memory
_monitor.erl, line 266)
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>       in call from vm_memory_monitor:internal_update/1 (vm_memory_monit
or.erl, line 409)
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>       in call from vm_memory_monitor:handle_info/2 (vm_memory_monitor.e
rl, line 230)
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>       in call from gen_server:try_dispatch/4 (gen_server.erl, line 695)
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>       in call from gen_server:handle_msg/6 (gen_server.erl, line 771)
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     ancestors: [vm_memory_monitor_sup,rabbit_sup,<0.221.0>]
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     message_queue_len: 0
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     messages: []
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     links: [<0.363.0>]
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     dictionary: []
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     trap_exit: false
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     status: running
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     heap_size: 17731
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     stack_size: 29
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>     reductions: 36285283
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0>   neighbours:
2023-08-10 05:36:16.470075+00:00 [error] <0.2181.0> 
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>     supervisor: {local,vm_memory_monitor_sup}
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>     errorContext: child_terminated
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>     reason: {{badmatch,{error,enfile}},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>              [{vm_memory_monitor,read_proc_file,1,
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                                  [{file,"vm_memory_monitor.erl"},{line,5
80}]},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>               {vm_memory_monitor,get_process_memory_using_strategy,2,
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                                  [{file,"vm_memory_monitor.erl"},{line,2
83}]},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>               {vm_memory_monitor,update_process_memory,1,
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                                  [{file,"vm_memory_monitor.erl"},{line,2
66}]},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>               {vm_memory_monitor,internal_update,1,
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                                  [{file,"vm_memory_monitor.erl"},{line,4
09}]},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>               {vm_memory_monitor,handle_info,2,
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                                  [{file,"vm_memory_monitor.erl"},{line,2
30}]},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>               {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,
695}]},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>               {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,77
1}]},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>               {proc_lib,init_p_do_apply,3,
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                         [{file,"proc_lib.erl"},{line,226}]}]}
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>     offender: [{pid,<0.2181.0>},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                {id,vm_memory_monitor},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                {mfargs,
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                    {vm_memory_monitor,start_link,
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                        [0.4,#Fun<rabbit_alarm.0.73210180>,
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                         #Fun<rabbit_alarm.1.73210180>]}},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                {restart_type,transient},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                {shutdown,300000},
2023-08-10 05:36:16.471238+00:00 [error] <0.363.0>                {child_type,worker}]
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>     supervisor: {local,vm_memory_monitor_sup}
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>     errorContext: start_error
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>     reason: {badarg,
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                 [{os,cmd,
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                      ["getconf PAGESIZE"],
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                      [{file,"os.erl"},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                       {line,419},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                       {error_info,#{module => erl_kernel_errors}}]},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                  {vm_memory_monitor,get_linux_pagesize,0,
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                      [{file,"vm_memory_monitor.erl"},{line,467}]},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                  {vm_memory_monitor,init_state_by_os,1,
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                      [{file,"vm_memory_monitor.erl"},{line,274}]},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                  {vm_memory_monitor,init,1,
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                      [{file,"vm_memory_monitor.erl"},{line,193}]},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                  {gen_server,init_it,2,[{file,"gen_server.erl"},{line,42
3}]},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                  {gen_server,init_it,6,[{file,"gen_server.erl"},{line,39
0}]},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                  {proc_lib,init_p_do_apply,3,
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                      [{file,"proc_lib.erl"},{line,226}]}]}
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>     offender: [{pid,<0.2181.0>},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                {id,vm_memory_monitor},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                {mfargs,
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                    {vm_memory_monitor,start_link,
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                        [0.4,#Fun<rabbit_alarm.0.73210180>,
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                         #Fun<rabbit_alarm.1.73210180>]}},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                {restart_type,transient},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                {shutdown,300000},
2023-08-10 05:36:16.475469+00:00 [error] <0.363.0>                {child_type,worker}]
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>   crasher:
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     initial call: vm_memory_monitor:init/1
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     pid: <0.13359.1>
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     registered_name: []
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     exception error: bad argument
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>       in function  os:cmd/1
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>          called as os:cmd("getconf PAGESIZE")
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>          *** argument 1: not a list of characters
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>       in call from vm_memory_monitor:get_linux_pagesize/0 (vm_memory_m
onitor.erl, line 467)
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>       in call from vm_memory_monitor:init_state_by_os/1 (vm_memory_mon
itor.erl, line 274)
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>       in call from vm_memory_monitor:init/1 (vm_memory_monitor.erl, li
ne 193)
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>       in call from gen_server:init_it/2 (gen_server.erl, line 423)
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>       in call from gen_server:init_it/6 (gen_server.erl, line 390)
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     ancestors: [vm_memory_monitor_sup,rabbit_sup,<0.221.0>]
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     message_queue_len: 0
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     messages: []
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     links: [<0.363.0>]
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     dictionary: []
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     trap_exit: false
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     status: running
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     heap_size: 1598
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     stack_size: 29
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>     reductions: 1546
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1>   neighbours:
2023-08-10 05:36:16.475959+00:00 [error] <0.13359.1> 
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>     supervisor: {local,vm_memory_monitor_sup}
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>     errorContext: start_error
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>     reason: {badarg,
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                 [{os,cmd,
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                      ["getconf PAGESIZE"],
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                      [{file,"os.erl"},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                       {line,419},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                       {error_info,#{module => erl_kernel_errors}}]},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                  {vm_memory_monitor,get_linux_pagesize,0,
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                      [{file,"vm_memory_monitor.erl"},{line,467}]},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                  {vm_memory_monitor,init_state_by_os,1,
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                      [{file,"vm_memory_monitor.erl"},{line,274}]},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                  {vm_memory_monitor,init,1,
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                      [{file,"vm_memory_monitor.erl"},{line,193}]},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                  {gen_server,init_it,2,[{file,"gen_server.erl"},{line,42
3}]},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                  {gen_server,init_it,6,[{file,"gen_server.erl"},{line,39
0}]},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                  {proc_lib,init_p_do_apply,3,
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                      [{file,"proc_lib.erl"},{line,226}]}]}
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>     offender: [{pid,{restarting,<0.2181.0>}},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                {id,vm_memory_monitor},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                {mfargs,
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                    {vm_memory_monitor,start_link,
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                        [0.4,#Fun<rabbit_alarm.0.73210180>,
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                         #Fun<rabbit_alarm.1.73210180>]}},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                {restart_type,transient},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                {shutdown,300000},
2023-08-10 05:36:16.478255+00:00 [error] <0.363.0>                {child_type,worker}]

I can not find out why. please help me!

[kmath163]

I try to print File Descripter get below content

lsof|grep beam.smp|wc -l
3840
lsof|grep rabbitmq|wc -l 
3879

[kmath163]

Erlang crash dump

sh erl_crashdump_analyzer.sh erl_crash.dump
-e analyzing erl_crash.dump, generated on:  Thu Aug 10 06:07:18 2023 

Slogan: Kernel pid terminated (logger) ({undef,[{maps,get,[filters,,[_]],[]},{logger_backend,log_allowed,3,[{_},{_}]},{proc_lib,exit_p,3,[{_},{_}]}]})
-e 
Memory:
===
erl_crashdump_analyzer.sh: 12: erl_crashdump_analyzer.sh: let: not found
  processes:  Mb
erl_crashdump_analyzer.sh: 15: erl_crashdump_analyzer.sh: let: not found
  processes_used:  Mb
erl_crashdump_analyzer.sh: 18: erl_crashdump_analyzer.sh: let: not found
  system:  Mb
erl_crashdump_analyzer.sh: 21: erl_crashdump_analyzer.sh: let: not found
  atom:  Mb
erl_crashdump_analyzer.sh: 24: erl_crashdump_analyzer.sh: let: not found
  atom_used:  Mb
erl_crashdump_analyzer.sh: 27: erl_crashdump_analyzer.sh: let: not found
  binary:  Mb
erl_crashdump_analyzer.sh: 30: erl_crashdump_analyzer.sh: let: not found
  code:  Mb
erl_crashdump_analyzer.sh: 33: erl_crashdump_analyzer.sh: let: not found
  ets:  Mb
erl_crashdump_analyzer.sh: 36: erl_crashdump_analyzer.sh: let: not found
-e   ---
  total:  Mb
-e 
Different message queue lengths (5 largest different):
===
     18 0
-e 
Error logger queue length:
===
-e 
File descriptors open:
===
-e   UDP:  0
-e   TCP:  0
-e   Files:  0
-e   ---
  Total:  0
-e 
Number of processes:
===
18
-e 
Processes Heap+Stack memory sizes (words) used in the VM (5 largest different):
===
      1 6772
      2 4185
      2 1598
      1 987
      2 376
-e 
Processes OldHeap memory sizes (words) used in the VM (5 largest different):
===
      1 46422
      1 6772
      1 4185
      1 2586
      1 1598
-e 
Process States when crashing (sum): 
===
      1 CONNECTED
      1 Current Process Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING
      1 Current Process Running
      3 Internal ACT_PRIO_HIGH | USR_PRIO_HIGH | PRQ_PRIO_HIGH | OFF_HEAP_MSGQ
      1 Internal ACT_PRIO_MAX | USR_PRIO_MAX | PRQ_PRIO_MAX | OFF_HEAP_MSGQ
     12 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL
      1 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING
      1 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | OFF_HEAP_MSGQ
      1 Running
     17 Waiting

[lukebakken]

Your issue description and comments are VERY DIFFICULT to read and understand. Please take the time to learn how GitHub markdown works - https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax

In addition, it's better to ATTACH large amounts of text as files rather than pasting them.

undef,[{maps,get suggests that the version of Erlang you are using is too old for the version of RabbitMQ you are using.