rust panic on Intel cpu when running windowpost

[mslovy]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

lotus-miner version 1.14.2+mainnet+git.a65386b49

Describe the Bug

running windowpost, rust panic on Once instance has previously been poisoned

Logging Information

2022-02-27T06:23:39.583Z        INFO    storageminer    storage/wdpost_run.go:646       computing window post   {"batch": 0, "elapsed": 188.357811965}
2022-02-27T06:23:39.583Z        ERROR   storageminer    storage/wdpost_run.go:648       error generating window post: Rust panic: Once instance has previously been poisoned
2022-02-27T06:23:39.583Z        ERROR   storageminer    storage/wdpost_run.go:105       runPoStCycle failed: running window post failed:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
        /lotus/storage/wdpost_run.go:709
  - Rust panic: Once instance has previously been poisoned
    github.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt
        /lotus/extern/filecoin-ffi/proofs.go:685
    github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt
        /lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:48
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
        /lotus/storage/wdpost_run.go:644
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runGeneratePoST
        /lotus/storage/wdpost_run.go:103
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).startGeneratePoST.func1
        /lotus/storage/wdpost_run.go:87
    runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371
2022-02-27T06:23:39.584Z        ERROR   storageminer    storage/wdpost_run.go:49        Got err running window post failed:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
        /lotus/storage/wdpost_run.go:709
  - Rust panic: Once instance has previously been poisoned
    github.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt
        /lotus/extern/filecoin-ffi/proofs.go:685
    github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt
        /lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:48
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
        /lotus/storage/wdpost_run.go:644
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runGeneratePoST
        /lotus/storage/wdpost_run.go:103
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).startGeneratePoST.func1
        /lotus/storage/wdpost_run.go:87
    runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371 - TODO handle errors
2022-02-27T06:23:39.584Z        WARN    storageminer    storage/wdpost_changehandler.go:255     Aborted window post Proving (Deadline: &{CurrentEpoch:1587881 PeriodStart:1587854 Index:0 Open:1587854 Close:1587914 Challenge:1587834 FaultCutoff:1587784 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})
2022-02-27T06:23:56.017Z        INFO    miner   miner/miner.go:484      completed mineOne       {"tookMilliseconds": 8, "forRound": 1587888, "baseEpoch": 1587887, "baseDeltaSeconds": 26, "nullRounds": 0, "lateStart": false, "beaconEpoch": 0, "lookbackEpochs": 900, "networkPowerAtLookback": "18043088970982326272", "minerPowerAtLookback": "0", "isEligible": false, "isWinner": false, "error": null}

Repo Steps

1. Run '...'
2. Do '...'
3. See error '...'
   ...

[Reiers]

Hi @mslovy
This seems hw related and not a Lotus issue.
Im moving this to Discussions for help and troubleshooting.

[seantw1974]

This is "HDD" issue, you have some sectors damaged.
Lotus-miner now cannot pass this damaged sectors and cause whole windowpost fail .
You may do as follow

1. Use "lotus-miner proving check --slow 0" to find out which sectors damaged, and remove it !!
2. Repate step one to check it
3. Use "sector-check" or "lotus-box" to emulate "windowpost"

One sector fail (vanilla proof fail )make whole partion fail already report issue here
#8286
You still have to check your sectors by manually this time