Rabbitmq 3.9.13 disk error nfs

[zoroglucihat]

Hi,
We have got an error on rabbitmq.

Error log

2022-11-22 03:38:52.720253+00:00 [info] <0.6160.4579> Enabling free disk space monitoring
2022-11-22 03:38:52.720333+00:00 [info] <0.6160.4579> Disk free limit set to 50MB
2022-11-23 11:05:51.076816+00:00 [warn] <0.6282.0> closing AMQP connection <0.6282.0> (192.168.7.165:37422 -> 192.168.6.69:5672 - connectionFactory#41f686af:0, vhost: '/', user: 'admin'):
2022-11-23 11:05:51.076816+00:00 [warn] <0.6282.0> client unexpectedly closed TCP connection
2022-11-23 11:06:51.186335+00:00 [warn] <0.6082.0> closing AMQP connection <0.6082.0> (192.168.8.160:50418 -> 192.168.6.69:5672 - connectionFactory.publisher#574e7d82:0, vhost: '/', user: 'admin'):
2022-11-23 11:06:51.186335+00:00 [warn] <0.6082.0> client unexpectedly closed TCP connection
2022-11-23 11:06:52.344508+00:00 [info] <0.13679.16> accepting AMQP connection <0.13679.16> (192.168.5.107:57370 -> 192.168.6.69:5672)
2022-11-23 11:06:52.346928+00:00 [info] <0.13679.16> Connection <0.13679.16> (192.168.5.107:57370 -> 192.168.6.69:5672) has a client-provided name: connectionFactory.publisher#1f60e4d5:0
2022-11-23 11:06:52.348160+00:00 [info] <0.13679.16> connection <0.13679.16> (192.168.5.107:57370 -> 192.168.6.69:5672 - connectionFactory.publisher#1f60e4d5:0): user 'admin' authenticated and granted access to vhost '/'
2022-11-23 11:07:11.135541+00:00 [warn] <0.6146.0> closing AMQP connection <0.6146.0> (192.168.4.208:42424 -> 192.168.6.69:5672 - connectionFactory.publisher#3d899262:0, vhost: '/', user: 'admin'):
2022-11-23 11:07:11.135541+00:00 [warn] <0.6146.0> client unexpectedly closed TCP connection
2022-11-24 03:36:28.193817+00:00 [erro] <0.6160.4579> Command timed out: '/bin/df -kP /bitnami/rabbitmq/mnesia/[email protected]'
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579> ** Generic server rabbit_disk_monitor terminating
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579> ** Last message in was update
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579> ** When Server state == {state,"/bitnami/rabbitmq/mnesia/[email protected]",
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>                                50000000,271151267840,100,10000,
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>                                #Ref<0.3415598053.3075473409.10798>,false,
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>                                false,9,120000}
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579> ** Reason for termination ==
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579> ** {function_clause,
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>        [{lists,reverse,[{error,timeout}],[{file,"lists.erl"},{line,147}]},
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>         {string,tokens,2,[{file,"string.erl"},{line,1934}]},
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>         {rabbit_disk_monitor,parse_free_unix,1,
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>             [{file,"rabbit_disk_monitor.erl"},{line,262}]},
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>         {rabbit_disk_monitor,internal_update,1,
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>             [{file,"rabbit_disk_monitor.erl"},{line,216}]},
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>         {rabbit_disk_monitor,handle_info,2,
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>             [{file,"rabbit_disk_monitor.erl"},{line,166}]},
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>         {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>         {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579>         {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2022-11-24 03:36:28.193953+00:00 [erro] <0.6160.4579> 
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>   crasher:
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     initial call: rabbit_disk_monitor:init/1
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     pid: <0.6160.4579>
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     registered_name: rabbit_disk_monitor
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     exception error: no function clause matching
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>                      lists:reverse({error,timeout}) (lists.erl, line 147)
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>       in function  string:tokens/2 (string.erl, line 1934)
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>       in call from rabbit_disk_monitor:parse_free_unix/1 (rabbit_disk_monitor.erl, line 262)
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>       in call from rabbit_disk_monitor:internal_update/1 (rabbit_disk_monitor.erl, line 216)
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>       in call from rabbit_disk_monitor:handle_info/2 (rabbit_disk_monitor.erl, line 166)
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>       in call from gen_server:try_dispatch/4 (gen_server.erl, line 695)
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>       in call from gen_server:handle_msg/6 (gen_server.erl, line 771)
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.228.0>]
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     message_queue_len: 0
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     messages: []
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     links: [<0.394.0>]
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     dictionary: []
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     trap_exit: false
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     status: running
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     heap_size: 10958
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     stack_size: 29
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>     reductions: 56595061
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579>   neighbours:
2022-11-24 03:36:28.194495+00:00 [erro] <0.6160.4579> 
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>     supervisor: {local,rabbit_disk_monitor_sup}
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>     errorContext: child_terminated
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>     reason: {function_clause,
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                 [{lists,reverse,
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                      [{error,timeout}],
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                      [{file,"lists.erl"},{line,147}]},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                  {string,tokens,2,[{file,"string.erl"},{line,1934}]},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                  {rabbit_disk_monitor,parse_free_unix,1,
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                      [{file,"rabbit_disk_monitor.erl"},{line,262}]},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                  {rabbit_disk_monitor,internal_update,1,
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                      [{file,"rabbit_disk_monitor.erl"},{line,216}]},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                  {rabbit_disk_monitor,handle_info,2,
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                      [{file,"rabbit_disk_monitor.erl"},{line,166}]},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                  {gen_server,try_dispatch,4,
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                      [{file,"gen_server.erl"},{line,695}]},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                  {gen_server,handle_msg,6,
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                      [{file,"gen_server.erl"},{line,771}]},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                  {proc_lib,init_p_do_apply,3,
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                      [{file,"proc_lib.erl"},{line,226}]}]}
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>     offender: [{pid,<0.6160.4579>},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                {id,rabbit_disk_monitor},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                {restart_type,{transient,1}},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                {shutdown,300000},
2022-11-24 03:36:28.195283+00:00 [erro] <0.394.0>                {child_type,worker}]

[lukebakken]

Here is the command that timed out:

Command timed out: '/bin/df -kP /bitnami/rabbitmq/mnesia/[email protected]'

More than likely your server is overloaded, or your storage system is.

This error will not affect RabbitMQ's normal operation.

[zoroglucihat]

Hi Luke, Thank you for your response. We are using helm bitnami. It is working on pod. And we are holding volumes on nfs server. Is that related with nfs server?
They said that we have experienced performance problem on rabbitmq the error occurance time.

[michaelklishin]

We very explicitly do not recommend storing RabbitMQ node data on network filesystems. Any slowdown of such network filesystem will severely affect virtually everything nodes do, in particular when quorum queues or streams are used.

[michaelklishin]

You did not "experience a RabbitMQ performance problem", you experienced a network filesystem slowdown or outage that did not result in any OS errors, which severely affected all operations performed by RabbitMQ, including free disk space monitoring.

[lukebakken]

we are using nfs server

As @michaelklishin said, do NOT do this. Use physically attached storage.

[zoroglucihat]

Thank you, @michaelklishin and @lukebakken