task journal-offline, blocked for more than 120 seconds.

[jxdking]

I get this message from the console couples times per day.
Is it caused by ZFS?
Should I worry about it?

[ 8459.295673] INFO: task journal-offline:17802 blocked for more than 120 seconds.
[ 8459.295721]       Tainted: P           O      5.15.0-67-generic #74-Ubuntu
[ 8459.295747] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8459.295774] task:journal-offline state:D stack:    0 pid:17802 ppid:  7235 flags:0x00000002
[ 8459.295779] Call Trace:
[ 8459.295780]  <TASK>
[ 8459.295783]  __schedule+0x24e/0x590
[ 8459.295789]  schedule+0x69/0x110
[ 8459.295791]  io_schedule+0x46/0x80
[ 8459.295793]  wait_on_page_bit_common+0x10c/0x3d0
[ 8459.295797]  ? filemap_invalidate_unlock_two+0x50/0x50
[ 8459.295799]  wait_on_page_bit+0x3f/0x50
[ 8459.295801]  wait_on_page_writeback+0x26/0x80
[ 8459.295804]  __filemap_fdatawait_range+0x97/0x120
[ 8459.295806]  filemap_write_and_wait_range+0x88/0xe0
[ 8459.295810]  zpl_fsync+0x3d/0xa0 [zfs]
[ 8459.295935]  vfs_fsync_range+0x46/0x90
[ 8459.295938]  ? __fget_light+0x39/0x90
[ 8459.295941]  __x64_sys_fsync+0x38/0x70
[ 8459.295943]  do_syscall_64+0x59/0xc0
[ 8459.295946]  ? syscall_exit_to_user_mode+0x27/0x50
[ 8459.295948]  ? syscall_enter_from_user_mode+0x2f/0x40
[ 8459.295951]  ? do_syscall_64+0x69/0xc0
[ 8459.295953]  ? switch_fpu_return+0x4e/0xc0
[ 8459.295956]  ? exit_to_user_mode_prepare+0x96/0xb0
[ 8459.295959]  ? syscall_exit_to_user_mode+0x27/0x50
[ 8459.295962]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 8459.295964] RIP: 0033:0x7f76cac22a5b
[ 8459.295966] RSP: 002b:00007f76c8229c50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[ 8459.295968] RAX: ffffffffffffffda RBX: 00005595b2c427a0 RCX: 00007f76cac22a5b
[ 8459.295970] RDX: 0000000000000000 RSI: 00007f76cafbe9f4 RDI: 000000000000001c
[ 8459.295971] RBP: 00007f76cafc0280 R08: 0000000000000000 R09: 00007ffd711de71f
[ 8459.295972] R10: 00007f76cab9b99a R11: 0000000000000293 R12: 0000000000000002
[ 8459.295973] R13: 0000000000000002 R14: 00007f76cab9b850 R15: 00007ffd711de760
[ 8459.295975]  </TASK>
[ 9305.116055] INFO: task journal-offline:18476 blocked for more than 120 seconds.
[ 9305.116084]       Tainted: P           O      5.15.0-67-generic #74-Ubuntu
[ 9305.116097] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9305.116111] task:journal-offline state:D stack:    0 pid:18476 ppid:     1 flags:0x00000002
[ 9305.116114] Call Trace:
[ 9305.116116]  <TASK>
[ 9305.116118]  __schedule+0x24e/0x590
[ 9305.116123]  schedule+0x69/0x110
[ 9305.116124]  io_schedule+0x46/0x80
[ 9305.116126]  wait_on_page_bit_common+0x10c/0x3d0
[ 9305.116129]  ? filemap_invalidate_unlock_two+0x50/0x50
[ 9305.116131]  wait_on_page_bit+0x3f/0x50
[ 9305.116132]  wait_on_page_writeback+0x26/0x80
[ 9305.116134]  __filemap_fdatawait_range+0x97/0x120
[ 9305.116136]  filemap_write_and_wait_range+0x88/0xe0
[ 9305.116139]  zpl_fsync+0x3d/0xa0 [zfs]
[ 9305.116223]  vfs_fsync_range+0x46/0x90
[ 9305.116225]  ? __fget_light+0x39/0x90
[ 9305.116228]  __x64_sys_fsync+0x38/0x70
[ 9305.116229]  do_syscall_64+0x59/0xc0
[ 9305.116231]  ? exit_to_user_mode_prepare+0x37/0xb0
[ 9305.116234]  ? syscall_exit_to_user_mode+0x27/0x50
[ 9305.116236]  ? __x64_sys_prctl+0x21/0x30
[ 9305.116238]  ? do_syscall_64+0x69/0xc0
[ 9305.116239]  ? sigprocmask+0xac/0xf0
[ 9305.116241]  ? exit_to_user_mode_prepare+0x37/0xb0
[ 9305.116243]  ? syscall_exit_to_user_mode+0x27/0x50
[ 9305.116244]  ? do_syscall_64+0x69/0xc0
[ 9305.116246]  ? syscall_exit_to_user_mode+0x27/0x50
[ 9305.116247]  ? syscall_enter_from_user_mode+0x2f/0x40
[ 9305.116249]  ? do_syscall_64+0x69/0xc0
[ 9305.116250]  ? syscall_exit_to_user_mode+0x27/0x50
[ 9305.116252]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 9305.116253] RIP: 0033:0x7f058a60fa5b
[ 9305.116255] RSP: 002b:00007f0589437c50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[ 9305.116257] RAX: ffffffffffffffda RBX: 000056111e035c70 RCX: 00007f058a60fa5b
[ 9305.116258] RDX: 0000000000000000 RSI: 00007f058a99f9f4 RDI: 000000000000001f
[ 9305.116259] RBP: 00007f058a9a1280 R08: 0000000000000000 R09: 00007ffdb3a9b03f
[ 9305.116259] R10: 00007f058a58899a R11: 0000000000000293 R12: 0000000000000002
[ 9305.116260] R13: 0000000000000002 R14: 00007f058a588850 R15: 00007ffdb3a9b080
[ 9305.116261]  </TASK>
[11117.588211] INFO: task journal-offline:19848 blocked for more than 120 seconds.
[11117.588244]       Tainted: P           O      5.15.0-67-generic #74-Ubuntu
[11117.588257] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[11117.588271] task:journal-offline state:D stack:    0 pid:19848 ppid:     1 flags:0x00000002
[11117.588274] Call Trace:
[11117.588275]  <TASK>
[11117.588278]  __schedule+0x24e/0x590
[11117.588283]  schedule+0x69/0x110
[11117.588285]  io_schedule+0x46/0x80
[11117.588286]  wait_on_page_bit_common+0x10c/0x3d0
[11117.588290]  ? filemap_invalidate_unlock_two+0x50/0x50
[11117.588292]  wait_on_page_bit+0x3f/0x50
[11117.588293]  wait_on_page_writeback+0x26/0x80
[11117.588295]  __filemap_fdatawait_range+0x97/0x120
[11117.588297]  filemap_write_and_wait_range+0x88/0xe0
[11117.588300]  zpl_fsync+0x3d/0xa0 [zfs]
[11117.588384]  vfs_fsync_range+0x46/0x90
[11117.588386]  ? __fget_light+0x39/0x90
[11117.588389]  __x64_sys_fsync+0x38/0x70
[11117.588390]  do_syscall_64+0x59/0xc0
[11117.588392]  ? switch_fpu_return+0x4e/0xc0
[11117.588395]  ? exit_to_user_mode_prepare+0x96/0xb0
[11117.588397]  ? syscall_exit_to_user_mode+0x27/0x50
[11117.588399]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[11117.588401] RIP: 0033:0x7f058a60fa5b
[11117.588403] RSP: 002b:00007f0589437c50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[11117.588405] RAX: ffffffffffffffda RBX: 000056111e035c70 RCX: 00007f058a60fa5b
[11117.588406] RDX: 0000000000000000 RSI: 00007f058a99f9f4 RDI: 000000000000001f
[11117.588406] RBP: 00007f058a9a1280 R08: 0000000000000000 R09: 00007ffdb3a9b03f
[11117.588407] R10: 00007f058a58899a R11: 0000000000000293 R12: 0000000000000002
[11117.588408] R13: 0000000000000002 R14: 00007f058a588850 R15: 00007ffdb3a9b080
[11117.588409]  </TASK>

[ryao]

It suggests that fsync took abnormally long to run. This could be caused by bad sectors on a hard drive. You might want to check the SMART data using smartctl. There are other possible causes, but that one is the most likely.

[jxdking]

I have slog device on all spinning pool. Slog should take care of fsync.
I migrated the system to AM5 platform less than a month, with new nvme device as slog. This error was also experienced in the old system. I tend to believe it is software issue. The system just spits out this error from time to time and stays functional, no crash. I have a windows vm running on top of zfs. I games on it daily, and it is absolutely fine. The only thing I can think of is the lxd containers. I experienced sometimes it hangs on lxc stop command. Let me check the if it is related.

[jxdking]

Reporting back.
It was caused by systemd-journald running in side lxd containers.
Right now, in journal.conf, I switched the "storage" to "volatile" in every lxd container. Since then, the error is gone.
It is still strange why systemd-journald would cause fsync hang, and all other applications on the pool run absolutely fine.