rabbitmq errors

[Biaoshi]

Describe the bug

some errors occur after running for days.

RabbitMQ 3.8.4
Erang 22.0.7

The error log contains the following information:

2024-03-28 15:55:14.075 [info] <0.31778.55> connection <0.31778.55> (10.190.5.190:48172 -> 10.190.5.248:5672): user 'account' authenticated and granted access to vhost '/'
2024-03-28 15:55:14.078 [info] <0.31778.55> closing AMQP connection <0.31778.55> (10.190.5.190:48172 -> 10.190.5.248:5672, vhost: '/', user: 'account')
2024-03-28 15:57:10.383 [error] <0.31774.55> closing AMQP connection <0.31774.55> (10.190.5.190:54288 -> 10.190.5.248:5672):
missed heartbeats from client, timeout: 60s
2024-03-28 15:57:11.580 [error] <0.31718.55> closing AMQP connection <0.31718.55> (10.190.5.190:54302 -> 10.190.5.248:5672):
missed heartbeats from client, timeout: 60s
2024-03-28 16:08:09.460 [error] <0.424.0> ** Generic server rabbit_disk_monitor terminating 
** Last message in was update
** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@centos",50000000,196627734528,100,10000,#Ref<0.504842236.897843201.49206>,false,true,10,120000}
** Reason for termination ==
** {{unparseable,[]},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2024-03-28 16:08:09.460 [error] <0.424.0> CRASH REPORT Process rabbit_disk_monitor with 0 neighbours exited with reason: {unparseable,[]} in rabbit_disk_monitor:parse_free_unix/1 line 272
2024-03-28 16:08:09.460 [error] <0.423.0> Supervisor rabbit_disk_monitor_sup had child rabbit_disk_monitor started with rabbit_disk_monitor:start_link(50000000) at <0.424.0> exit with reason {unparseable,[]} in context child_terminated
2024-03-28 16:08:09.461 [info] <0.82.56> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,[]}},25110732800}, retries left: 10
2024-03-28 16:08:11.268 [error] <0.11627.21> closing AMQP connection <0.11627.21> (10.190.5.252:53632 -> 10.190.5.248:5672 - createFactory#4e3e11b9:957936):
missed heartbeats from client, timeout: 1s
=============================================
========================================================
The  crash.log.0  contains the following information:

2024-03-28 16:08:09 =ERROR REPORT====
** Generic server rabbit_disk_monitor terminating 
** Last message in was update
** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@centos",50000000,196627734528,100,10000,#Ref<0.504842236.897843201.49206>,false,true,10,120000}
** Reason for termination ==
** {{unparseable,[]},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2024-03-28 16:08:09 =CRASH REPORT====
  crasher:
    initial call: rabbit_disk_monitor:init/1
    pid: <0.424.0>
    registered_name: rabbit_disk_monitor
    exception exit: {{unparseable,[]},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
    ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.263.0>]
    message_queue_len: 0
    messages: []
    links: [<0.423.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 200744405
  neighbours:
2024-03-28 16:08:09 =SUPERVISOR REPORT====
     Supervisor: {local,rabbit_disk_monitor_sup}
     Context:    child_terminated
     Reason:     {unparseable,[]}
     Offender:   [{pid,<0.424.0>},{name,rabbit_disk_monitor},{mfargs,{rabbit_disk_monitor,start_link,[50000000]}},{restart_type,{transient,1}},{shutdown,30000},{child_type,worker}]

Reproduction steps

An error occurs when the program is running normally

Expected behavior

I want to know how to solve this problem or avoid it

Additional context

No response

[michaelklishin]

RabbitMQ 3.8 has reached EOL several years ago. Even 3.10 is out of support, and 3.11.x will reach EOL in a few months.

The error says that a free disk space monitoring has failed to parse the output from df -k. All this means is that a single node metric (the amount of free disk space available) won't be available, the node should still be up and running even then.

There have been numerous updates to the free disk space monitor parser over the years.

[Biaoshi]

So should I upgrade my rabbitmq to a later version? Is there a recommended version? My Jdk version is 1.8

[michaelklishin]

Not only you should, you must because the version you use is COMPLETELY out of support. The recommended version is the latest there is. 3.13.1 may ship very soon but learning about upgrading to 3.13.0 would be a good start.