[Sealing Issue] panicked at 'Once instance has previously been poisoned'

[threadfly]

Note: For security-related bugs/issues, please follow the security policy.

The following is the stack when lotus worker crashes：
2021-07-14T10:58:10.890+0800 DEBUG advmgr sector-storage/worker_local.go:129 acquired sector {{5561 0} 8} (e:1; a:6): {{0 0} /data/tmp_sealing/unsealed/s-t05561-0 /data/tmp_sealing/sealed/s-t05561-0 /data/tmp_sealing/cache/s-t05561-0}
thread '' panicked at 'Once instance has previously been poisoned', library/std/src/sync/once.rs:392:21
stack backtrace:
0: 0x229ca00 - std::backtrace_rs::backtrace::libunwind::trace::hdcf4f90f85129e83
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
1: 0x229ca00 - std::backtrace_rs::backtrace::trace_unsynchronized::h2669e30cb82f6732
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x229ca00 - std::sys_common::backtrace::_print_fmt::hfbda19e17f6db318
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:67:5
3: 0x229ca00 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h1a8751bf59281272
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:46:22
4: 0x230cc6f - core::fmt::write::h7aa6cd0067dca82a
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/core/src/fmt/mod.rs:1094:17
5: 0x2291195 - std::io::Write::write_fmt::hd7dd3a1df9b6befb
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/io/mod.rs:1580:15
6: 0x22a075b - std::sys_common::backtrace::_print::h551e9ec8a9fa8106
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:49:5
7: 0x22a075b - std::sys_common::backtrace::print::ha4b1c5e95fa040b3
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:36:9
8: 0x22a075b - std::panicking::default_hook::{{closure}}::h0b34c9ab7fb9f857
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:208:50
9: 0x22a023d - std::panicking::default_hook::h3067e8318decd17a
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:225:9
10: 0x22a0e4d - std::panicking::rust_panic_with_hook::h81b8facc50f34daa
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:591:17
11: 0x22a0c84 - std::panicking::begin_panic::{{closure}}::h7d6f9356b222c9cc
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:520:9
12: 0x229ce9c - std::sys_common::backtrace::__rust_end_short_backtrace::h52c52180b80b2652
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:141:18
13: 0x53f38c - std::panicking::begin_panic::h6d1bc5edb1568721
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:519:12
14: 0x53f179 - std::sync::once::Once::call_inner::hcf7ef963f95f83ca
at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sync/once.rs:392:21
15: 0x20352bf - storage_proofs_porep::stacked::vanilla::cores::checkout_core_group::hdcd3ed9d39569056
16: 0x1912177 - storage_proofs_porep::stacked::vanilla::create_label::multi::create_labels_for_encoding::h6b024bad3b136668
17: 0x18dc962 - storage_proofs_porep::stacked::vanilla::proof::StackedDrg<Tree,G>::generate_labels_for_encoding::hd605db7103302df5
18: 0x18c2788 - storage_proofs_porep::stacked::vanilla::proof::StackedDrg<Tree,G>::replicate_phase1::hcbd0f46b43b17195
19: 0x1953393 - filecoin_proofs::api::seal::seal_pre_commit_phase1::h3f51dc4b04d0ecf2
20: 0x187ecae - filecoin_proofs_api::seal::seal_pre_commit_phase1_inner::h03e67308e5c21d96
21: 0x187c7b5 - filecoin_proofs_api::seal::seal_pre_commit_phase1::hbb1c2312b2783e99
22: 0x1a11f9e - <std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once::h4f04a572e157bf5d
23: 0x1836f67 - ffi_toolkit::catch_panic_response::h51eaa5f8ef2f3cac
24: 0x1acb33e - fil_seal_pre_commit_phase1
25: 0x1776830 - _cgo_c8a539579dfc_Cfunc_fil_seal_pre_commit_phase1
at /tmp/go-build/cgo-gcc-prolog:1211:11
26: 0x5c43f0 - runtime.asmcgocall
at /usr/local/go/src/runtime/asm_amd64.s:656
2021-07-14T11:01:15.031+0800 DEBUG advmgr sector-storage/worker_local.go:129 acquired sector {{5561 0} 8} (e:1; a:0): {{0 0} /data/tmp_sealing/unsealed/s-t05561-0 }

Describe the problem
lotus worker crashes when seal sectors

Version

lotus version 1.10.0+calibnet+git.764fa9dae.dirty

[Reiers]

Sending this ticket to Lotus Discussions for troubleshooting.

[NiwanDao]

Any update on this ? @Reiers

[LexLuthr]

@xingjitansuo Are you running into a similar issue?

The once instance is poisoned when a thread fails while the lock is held. I think this is RwLock on your sector.
https://doc.rust-lang.org/std/sync/struct.PoisonError.html

If you are running into a similar issue, try to restart the concerned worker and check if the error goes away. Restart should release all locks and ffi should be able to get a lock on the unsealed sector to seal it on the next run.

Please let me know how it goes as this might help improve the product.

[LexLuthr]

I am archiving this discussion as there has been no new update from the author for more than 7 days.