generate window post skipped sectors

[KPWorkspace]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

dzd@dzd-s-101:/mnt/data/log$ lotus version
Daemon:  1.13.0+mainnet+git.7a55e8e89+api1.4.0
Local: lotus version 1.13.0+mainnet+git.7a55e8e89
dzd@dzd-s-101:/mnt/data/log$ lotus-miner version
Daemon:  1.13.0+mainnet+git.7a55e8e89.dirty+api1.2.0
Local: lotus-miner version 1.13.0+mainnet+git.7a55e8e89.dirty

Describe the Bug

My miners have several sectors of windowPost fault every day.

dzd@dzd-s-101:/mnt/data/log$ lotus-miner proving faults
deadline  partition  sectors
6         0          11849
7         0          17715
7         0          17875
7         0          18303
7         0          18525
7         0          18716
7         0          18860
dzd@dzd-s-101:/mnt/data/log$ lotus-miner proving check --slow --only-bad 6
deadline  partition  sector  status
6         0          12024   bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(12024)

Caused by:
    Generated vanilla proof for sector SectorId(12024) is invalid

Stack backtrace:
   0: anyhow::private::new_adhoc
   1: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   2: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   3: <rayon::iter::fold::FoldFolder<C,ID,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: std::panicking::try
   6: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
   7: rayon_core::registry::WorkerThread::wait_until_cold
   8: rayon_core::registry::ThreadBuilder::run
   9: std::sys_common::backtrace::__rust_begin_short_backtrace
  10: core::ops::function::FnOnce::call_once{{vtable.shim}}
  11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys/unix/thread.rs:71:17
  12: start_thread
  13: __clone)
dzd@dzd-s-101:/mnt/data/log$ lotus-miner proving check --slow --only-bad 7
deadline  partition  sector  status
7         0          18201   bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(18201)

Caused by:
    Generated vanilla proof for sector SectorId(18201) is invalid

Stack backtrace:
   0: anyhow::private::new_adhoc
   1: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   2: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   3: <rayon::iter::fold::FoldFolder<C,ID,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: rayon_core::join::join_context::{{closure}}
   6: rayon::iter::plumbing::bridge_producer_consumer::helper
   7: std::panicking::try
   8: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
   9: rayon_core::registry::WorkerThread::wait_until_cold
  10: rayon_core::registry::ThreadBuilder::run
  11: std::sys_common::backtrace::__rust_begin_short_backtrace
  12: core::ops::function::FnOnce::call_once{{vtable.shim}}
  13: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys/unix/thread.rs:71:17
  14: start_thread
  15: __clone)
7  0  18303  bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(18303)

Caused by:
    Generated vanilla proof for sector SectorId(18303) is invalid

Stack backtrace:
   0: anyhow::private::new_adhoc
   1: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   2: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   3: <rayon::iter::fold::FoldFolder<C,ID,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: rayon_core::join::join_context::{{closure}}
   6: rayon::iter::plumbing::bridge_producer_consumer::helper
   7: rayon_core::join::join_context::{{closure}}
   8: rayon::iter::plumbing::bridge_producer_consumer::helper
   9: std::panicking::try
  10: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  11: rayon_core::registry::WorkerThread::wait_until_cold
  12: rayon_core::registry::ThreadBuilder::run
  13: std::sys_common::backtrace::__rust_begin_short_backtrace
  14: core::ops::function::FnOnce::call_once{{vtable.shim}}
  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys/unix/thread.rs:71:17
  16: start_thread
  17: __clone)
7  0  18097  bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(18097)

Caused by:
    Generated vanilla proof for sector SectorId(18097) is invalid

Stack backtrace:
   0: anyhow::private::new_adhoc
   1: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   2: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   3: <rayon::iter::fold::FoldFolder<C,ID,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: std::panicking::try
   6: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
   7: rayon_core::registry::WorkerThread::wait_until_cold
   8: rayon_core::registry::ThreadBuilder::run
   9: std::sys_common::backtrace::__rust_begin_short_backtrace
  10: core::ops::function::FnOnce::call_once{{vtable.shim}}
  11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys/unix/thread.rs:71:17
  12: start_thread
  13: __clone)
dzd@dzd-s-101:/mnt/data/log$ lotus-miner proving check --slow --only-bad 6
deadline  partition  sector  status
6         0          12384   bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(12384)

Caused by:
    Generated vanilla proof for sector SectorId(12384) is invalid

Stack backtrace:
   0: anyhow::private::new_adhoc
   1: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   2: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   3: <rayon::iter::fold::FoldFolder<C,ID,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: rayon_core::join::join_context::{{closure}}
   6: rayon::iter::plumbing::bridge_producer_consumer::helper
   7: rayon_core::join::join_context::{{closure}}
   8: rayon::iter::plumbing::bridge_producer_consumer::helper
   9: rayon_core::join::join_context::{{closure}}
  10: std::panicking::try
  11: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  12: rayon_core::registry::WorkerThread::wait_until_cold
  13: rayon_core::registry::ThreadBuilder::run
  14: std::sys_common::backtrace::__rust_begin_short_backtrace
  15: core::ops::function::FnOnce::call_once{{vtable.shim}}
  16: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys/unix/thread.rs:71:17
  17: start_thread
  18: __clone)
6  0  12834  bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(12834)

Caused by:
    Generated vanilla proof for sector SectorId(12834) is invalid

Stack backtrace:
   0: anyhow::private::new_adhoc
   1: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   2: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   3: <rayon::iter::fold::FoldFolder<C,ID,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: rayon_core::join::join_context::{{closure}}
   6: rayon::iter::plumbing::bridge_producer_consumer::helper
   7: std::panicking::try
   8: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
   9: rayon_core::registry::WorkerThread::wait_until_cold
  10: rayon_core::registry::ThreadBuilder::run
  11: std::sys_common::backtrace::__rust_begin_short_backtrace
  12: core::ops::function::FnOnce::call_once{{vtable.shim}}
  13: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys/unix/thread.rs:71:17
  14: start_thread
  15: __clone)
dzd@dzd-s-101:/mnt/data/log$ lotus-miner proving check --slow --only-bad 6
deadline  partition  sector  status
6         0          11849   bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(11849)

Caused by:
    Generated vanilla proof for sector SectorId(11849) is invalid

Stack backtrace:
   0: anyhow::private::new_adhoc
   1: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   2: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   3: <rayon::iter::fold::FoldFolder<C,ID,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: rayon_core::join::join_context::{{closure}}
   6: rayon::iter::plumbing::bridge_producer_consumer::helper
   7: std::panicking::try
   8: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
   9: rayon_core::registry::WorkerThread::wait_until_cold
  10: rayon_core::registry::ThreadBuilder::run
  11: std::sys_common::backtrace::__rust_begin_short_backtrace
  12: core::ops::function::FnOnce::call_once{{vtable.shim}}
  13: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/alloc/src/boxed.rs:1546:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys/unix/thread.rs:71:17
  14: start_thread
  15: __clone)

Logging Information

more deal-miner-2021-12-22-17-39.log | grep -i -E 'wdpost|window_post'
2021-12-30T04:12:34.953+0800  WARN  storageminer  storage/wdpost_run.go:697 generate window post skipped sectors  {"sectors": [{"Miner":1350631,"Number":11849}], "error": "faulty sectors [SectorId(11849)]

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: <storage_proofs_post::fallback::vanilla::FallbackPoSt<Tree> as storage_proofs_core::proof::ProofScheme>::prove_all_partitions
   2: storage_proofs_core::compound_proof::CompoundProof::prove
   3: filecoin_proofs::api::window_post::generate_window_post
   4: filecoin_proofs_api::post::generate_window_post_inner
   5: filecoin_proofs_api::post::generate_window_post
   6: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
   7: ffi_toolkit::catch_panic_response
   8: fil_generate_window_post
   9: _cgo_c8a539579dfc_Cfunc_fil_generate_window_post
             at /tmp/go-build/cgo-gcc-prolog:645
  10: runtime.asmcgocall
             at /usr/local/go/src/runtime/asm_amd64.s:667", "errorVerbose": "faulty sectors [SectorId(11849)]

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: <storage_proofs_post::fallback::vanilla::FallbackPoSt<Tree> as storage_proofs_core::proof::ProofScheme>::prove_all_partitions
   2: storage_proofs_core::compound_proof::CompoundProof::prove
   3: filecoin_proofs::api::window_post::generate_window_post
   4: filecoin_proofs_api::post::generate_window_post_inner
   5: filecoin_proofs_api::post::generate_window_post
   6: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
   7: ffi_toolkit::catch_panic_response
   8: fil_generate_window_post
   9: _cgo_c8a539579dfc_Cfunc_fil_generate_window_post
             at /tmp/go-build/cgo-gcc-prolog:645
  10: runtime.asmcgocall
             at /usr/local/go/src/runtime/asm_amd64.s:667
             github.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt
              /home/dzd/workspace/lotus/extern/filecoin-ffi/proofs.go:684
github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt
  /home/dzd/workspace/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:46
github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
  /home/dzd/workspace/lotus/storage/wdpost_run.go:639
github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runGeneratePoST
  /home/dzd/workspace/lotus/storage/wdpost_run.go:103
github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).startGeneratePoST.func1
  /home/dzd/workspace/lotus/storage/wdpost_run.go:87
runtime.goexit
  /usr/local/go/src/runtime/asm_amd64.s:1371", "try": 0}

Repo Steps

What can I do to avoid punishment, please help me

[Reiers]

Hi @KPWorkspace

Thanks for the report. The issue you are experiencing lies elsewhere.

Moving this issue to Lotus Discussions for help and troubleshooting.

Closing Lotus Issue ticket.

[Reiers]

I noticed that you running on .dirty version.
1.13.0+mainnet+git.7a55e8e89.dirty+api1.2.0

I will suggest that build Lotus again and upgrade Lotus to latest release:
https://github.com/filecoin-project/lotus/releases/tag/v1.13.1

Build with CUDA:

cd lotus/
git pull
git checkout v1.13.1
git submodule update
RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA=1 make clean all
sudo make install

Build with OpenCL:

cd lotus/
git pull
git checkout v1.13.1
git submodule update
RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make clean all
sudo make install

Troubleshooting steps:

• Check your connection between your miner and your long term storage.
• Run a lotus-miner proving check --slow 0 (or 1,2.. whatever deadline that have faults)
• Check your nvidia driver, and its not on a bad driver. Here's a list: Overview of Nvidia Drivers for Lotus community#172
• nvidia-smi for checking if GPU is installed correctly
• Double check your dependencies:
  sudo apt update -y && sudo apt install -y nvidia-opencl-icd -y
sudo apt install mesa-opencl-icd ocl-icd-opencl-dev gcc git bzr jq pkg-config curl clang build-essential hwloc libhwloc-dev wget -y && sudo apt upgrade -y

Rustup:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Go:
wget -c https://golang.org/dl/go1.16.4.linux-amd64.tar.gz -O - | sudo tar -xz -C /usr/local
go --version

• It doesn't look like a sync error, doesn't hurt to check.
• Check that .lotus .lotusminer path is not full.

Please follow the troubleshooting steps and see if that fixes it or you find what cause the problem.
If the issue still persist - comment, and we will continue to troubleshoot.

[KPWorkspace]

Thanks, I will try your suggestion

[KPWorkspace]

Hello, after a few days of trying, there is still such a problem. Every time I run 'lotus-miner proving check --slow --only-bad 5', the sectors that report errors are different, as follows

kp@Ubuntu:~$ lotus-miner proving faults
deadline  partition  sectors
5         0          16273
7         0          16906
7         0          16922
7         0          17572
7         0          17751
7         0          18097
kp@Ubuntu:~$ lotus-miner proving check --slow --only-bad 5
deadline  partition  sector  status
5         0          14367   bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(14367)

Caused by:
    Generated vanilla proof for sector SectorId(14367) is invalid

Stack backtrace:
   0: anyhow::private::new_adhoc
   1: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   2: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   3: <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: std::panicking::try
   6: rayon_core::registry::in_worker
   7: rayon::iter::plumbing::bridge_producer_consumer::helper
   8: std::panicking::try
   9: rayon_core::registry::in_worker
  10: rayon::iter::plumbing::bridge_producer_consumer::helper
  11: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  12: std::panicking::try
  13: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  14: rayon_core::registry::WorkerThread::wait_until_cold
  15: rayon_core::registry::ThreadBuilder::run
  16: std::sys_common::backtrace::__rust_begin_short_backtrace
  17: core::ops::function::FnOnce::call_once{{vtable.shim}}
  18: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/b70888601af92f6cdc0364abab3446e418b91d36/library/alloc/src/boxed.rs:1572:9
  19: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/b70888601af92f6cdc0364abab3446e418b91d36/library/alloc/src/boxed.rs:1572:9
  20: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/b70888601af92f6cdc0364abab3446e418b91d36/library/std/src/sys/unix/thread.rs:91:17
  21: start_thread
  22: __clone)
kp@Ubuntu:~$ lotus-miner proving check --slow --only-bad 5
deadline  partition  sector  status
kp@Ubuntu:~$ lotus-miner proving check --slow --only-bad 5
deadline  partition  sector  status
5         0          16503   bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(16503)

Caused by:
    Generated vanilla proof for sector SectorId(16503) is invalid

Stack backtrace:
   0: anyhow::private::new_adhoc
   1: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   2: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   3: <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: std::panicking::try
   6: rayon_core::registry::in_worker
   7: rayon::iter::plumbing::bridge_producer_consumer::helper
   8: std::panicking::try
   9: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  10: rayon_core::registry::WorkerThread::wait_until_cold
  11: rayon_core::registry::ThreadBuilder::run
  12: std::sys_common::backtrace::__rust_begin_short_backtrace
  13: core::ops::function::FnOnce::call_once{{vtable.shim}}
  14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/b70888601af92f6cdc0364abab3446e418b91d36/library/alloc/src/boxed.rs:1572:9
  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/b70888601af92f6cdc0364abab3446e418b91d36/library/alloc/src/boxed.rs:1572:9
  16: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/b70888601af92f6cdc0364abab3446e418b91d36/library/std/src/sys/unix/thread.rs:91:17
  17: start_thread
  18: __clone)
5  0  14853  bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(14853)

Caused by:
    Generated vanilla proof for sector SectorId(14853) is invalid

Stack backtrace:
   0: anyhow::private::new_adhoc
   1: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   2: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   3: <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: std::panicking::try
   6: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
   7: rayon_core::registry::WorkerThread::wait_until_cold
   8: rayon_core::registry::ThreadBuilder::run
   9: std::sys_common::backtrace::__rust_begin_short_backtrace
  10: core::ops::function::FnOnce::call_once{{vtable.shim}}
  11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/b70888601af92f6cdc0364abab3446e418b91d36/library/alloc/src/boxed.rs:1572:9
  12: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/b70888601af92f6cdc0364abab3446e418b91d36/library/alloc/src/boxed.rs:1572:9
  13: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/b70888601af92f6cdc0364abab3446e418b91d36/library/std/src/sys/unix/thread.rs:91:17
  14: start_thread
  15: __clone)
kp@Ubuntu:~$ 

I don't know what causes this error!