Invalid encoding proof generated at layer 1

[ssim-itaz]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

Daemon:  1.15.3-dev+mainnet+git.33a05d33a+api1.5.0
Local: lotus-miner version 1.15.3-dev+mainnet+git.33a05d33a

Describe the Bug

thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread 'thread '' panicked at '' panicked at 'Invalid encoding proof generated at layer 1Invalid encoding proof generated at layer 1', ', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs/root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs::261261:thread ':3333
' panicked at '
Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread 'thread '' panicked at '' panicked at 'Invalid encoding proof generated at layer 1Invalid encoding proof generated at layer 1', ', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs/root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs::261261::3333

thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:thread '261:' panicked at '33Invalid encoding proof generated at layer 1
', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
thread '' panicked at 'Invalid encoding proof generated at layer 1', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-11.0.2/src/stacked/vanilla/proof.rs:261:33
2022-05-24T06:09:55.468Z WARN ffiwrapper ffiwrapper/sealer_cgo.go:740 checking PreCommit failed: Rust panic: no unwind information
2022-05-24T06:09:55.468Z WARN ffiwrapper ffiwrapper/sealer_cgo.go:741 num:19919 tkt:[87 118 126 54 45 98 10 33 189 190 90 84 193 39 165 242 34 73 79 46 75 43 19 75 1 37 102 163 198 242 206 59] seed:[242 138 103 247 171 117 198 0 42 206 130 147 56 150 134 5 254 121 246 145 155 58 141 95 196 174 208 87 208 174 3 65] sealedCID:bagboea4b5abcarzhw2w2mbrsuvatnojb7r4lwg5hgax2y5ceung2hik5rodcc23n, unsealedCID:baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq

Logging Information

precommit2 error

Repo Steps

1. Run '...'
2. Do '...'
3. See error '...'
   ...

[rjan90]

Hey @ssim-itaz! Since Rust panic: no unwind information is usually caused by bad graphics driver or bad ram. I am moving this to the Github discussion so we can investigate more. If we find out that there is an issue with Lotus we will move it back into the issue section.

I have a couple of questions:
What kind of Nvidia-driver are you using?
And are you using OpenCL or CUDA?
Can you run a Memtest? And are you using ECC RAM?

[ssim-itaz]

What kind of Nvidia-driver are you using?
-> nvidia-driver-510, cuda-toolkit-11-7, cuda-drivers-510
And are you using OpenCL or CUDA?
-> using CUDA
Can you run a Memtest? And are you using ECC RAM?
-> Yes memory, ssd, gpu, cpu test OK
and using ECC-RD RAM

[rjan90]

Okay, good! I see that you also are running the master-branch, can you please revert to a more stable release like v1.15.2 or v1.15.3-rc1.

[ssim-itaz]

I will test it after changing to version 1.15.2

[rjan90]

Nice 👍

[ssim-itaz]

Even after installing '1.15.2', the same error occurred.
I reinstalled the 'OS, 1.15.3, CUDA, RUST, GO'. 'No problem.

[ssim-itaz]

I think the 'rust version' is the problem.
My server install 'rust version' is 1.61, and the version downloaded when compiling lotus is '1.58'.

[beck-8]

The log is more detailed,lotus-miner sectors status --log xx
There are only three reasons for the problem

• parent_cache is error
• pc2 tasks was killed at running tree r
• gpu driver or gpu hardware has problem

Trust me, you will find the answer