One sector vanilla_proof fault(Due to Physical HD error) cause all partition fauls, that's not right thing !!! (vanilla_proof)

[seantw1974]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

1.15.x 
ALL version

Describe the Bug

I got some disk error !
And I use lotus-miner proving check to find the issue ..
I found If I only have one sector error (due to hardware issue)
But this cuase all my partion proof faults ..
The true is I only have one sector fault , but I have to accept all partion sectors error ????
This is not right thing !!!

Logging Information

I got some disk error !
And I use lotus-miner proving check to find the issue ..
I found If I only have one sector error (due to hardware issue)
But this cuase all my partion proof faults ..
The true is I only have one sector fault , but I have to accept all partion sectors error ????
This is not right thing !!!
You should add try {} catch {} function inside !

Repo Steps

I got some disk error !
And I use lotus-miner proving check to find the issue ..
I found If I only have one sector error (due to hardware issue)
But this cuase all my partion proof faults ..
The true is I only have one sector fault , but I have to accept all partion sectors error ????
This is not right thing !!!
You should add try {} catch {} function inside !

[Reiers]

@seantw1974 Hi

First glance - This looks like its hardware related and not a Lotus (software) issue.
There are no logs here, and nothing for us to reproduce.

So I will move it to our Discussions for help and troubleshooting.

If we find out that is something wrong in code itself then I will move it back to issues.

[seantw1974]

It's a common issue !!
Hardware issue may cause several sector fail, but your code make "WHOLE Partiton" fail...
Its because when "Rust panic .." happen, your code not skip error sector !!!
One Sector fail make Whole Partion fail , this is what your cod doing ..

[seantw1974]

lotus-miner proving check --slow can find issue sector (skip error) and work fine !
but why windowpost post cannot skip panic error make whole partion fail ??

[seantw1974]

When windowpost , your code just call fn "generate_window_post" and leave it alone ...
"Proving check --slow" call fn "generate_single_vanilla_proof" to check status ..
"generate_window_post" real code inside "RUST" and "filcoin_ffi" not in "lotus"
The real code inside "Filecoin_FFi=> genenrate_window_post or fi_generate_window_post " NO TRY/CATCH!!!
but you still have to find way to add TRY/CATCH

In "Filscan" website, you can find many many sectors fail, most of them have such issue ~
The code logic is wrong

[Reiers]

Please set adjust export RUST_LOG=debug for more logs to share, so we can have a look.

Troubleshooting:

• What kind of storage solution are you using?
  Is it undergoing any maintenance after the harddrive failure - because that can result in fault the whole deadline.

Also:
This will cmd will check each sector:
lotus-miner proving check --slow
Also check the time.

Theres also this tool you can use to emulate a windowPost while troubleshooting.
https://github.com/TippyFlitsUK/lotus-box.git

git clone https://github.com/TippyFlitsUK/lotus-box.git
here is cmd to run
export FULLNODE_API_INFO=key:/ip4/<ip-address>/tcp/1234/http \
; export FIL_PROOFS_PARAMETER_CACHE=/<path>/.lotusproofs \
; export FIL_PROOFS_PARENT_CACHE=/<path>/.lotusproofs \
; export BELLMAN_CUSTOM_GPU="<gpu>" \
; ./lotus-wdpsot d-emulator --deadline <deadline-id> --sdir /<storage path> --actor <miner-id>

[seantw1974]

This command I use many time ... "lotus-miner proving check --slow 0 " Indid one or more sectors fault.. but but but ..this cause whole partition Fail!!!!!! Why ? Why caue WHOLE PARTITION!! You should add try {} catch {} to prevent panic !!!!! Nicklas ***@***.***> 於 2022年3月10日 週四 上午2:52寫道：

…

Please set adjust export RUST_LOG=debug for more logs to share, so we can have a look. Troubleshooting: - What kind of storage solution are you using? Is it undergoing any maintenance after the harddrive failure - because that can result in fault the whole deadline. Also: This will cmd will check each sector: lotus-miner proving check --slow *Also check the time.* Theres also this tool you can use to emulate a windowPost while troubleshooting. https://github.com/TippyFlitsUK/lotus-box.git git clone https://github.com/TippyFlitsUK/lotus-box.git here is cmd to run export FULLNODE_API_INFO=key:/ip4/<ip-address>/tcp/1234/http \ ; export FIL_PROOFS_PARAMETER_CACHE=/<path>/.lotusproofs \ ; export FIL_PROOFS_PARENT_CACHE=/<path>/.lotusproofs \ ; export BELLMAN_CUSTOM_GPU="<gpu>" \ ; ./lotus-wdpsot d-emulator --deadline <deadline-id> --sdir /<storage path> --actor <miner-id> — Reply to this email directly, view it on GitHub <#8274 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVTHC2FBUW7XGEFRZLE5S3U7DXOBANCNFSM5QKNF2AA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[seantw1974]

I also use "sector-check" can emu wdpost At some kind of issue will "STOP" wdpost ... again you should ADD TRY CATCH BLOCK！！！！ not make WHOLE partition fail Cheng Yuan Chang ***@***.***> 於 2022年3月10日 週四 下午6:38寫道：

…

This command I use many time ... "lotus-miner proving check --slow 0 " Indid one or more sectors fault.. but but but ..this cause whole partition Fail!!!!!! Why ? Why caue WHOLE PARTITION!! You should add try {} catch {} to prevent panic !!!!! Nicklas ***@***.***> 於 2022年3月10日 週四 上午2:52寫道： > Please set adjust export RUST_LOG=debug for more logs to share, so we > can have a look. > > Troubleshooting: > > - What kind of storage solution are you using? > Is it undergoing any maintenance after the harddrive failure - > because that can result in fault the whole deadline. > > Also: > This will cmd will check each sector: > lotus-miner proving check --slow > *Also check the time.* > > Theres also this tool you can use to emulate a windowPost while > troubleshooting. > https://github.com/TippyFlitsUK/lotus-box.git > > git clone https://github.com/TippyFlitsUK/lotus-box.git > here is cmd to run > export FULLNODE_API_INFO=key:/ip4/<ip-address>/tcp/1234/http \ > ; export FIL_PROOFS_PARAMETER_CACHE=/<path>/.lotusproofs \ > ; export FIL_PROOFS_PARENT_CACHE=/<path>/.lotusproofs \ > ; export BELLMAN_CUSTOM_GPU="<gpu>" \ > ; ./lotus-wdpsot d-emulator --deadline <deadline-id> --sdir /<storage path> --actor <miner-id> > > — > Reply to this email directly, view it on GitHub > <#8274 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AEVTHC2FBUW7XGEFRZLE5S3U7DXOBANCNFSM5QKNF2AA> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > You are receiving this because you were mentioned.Message ID: > ***@***.*** > com> >

[Reiers]

@seantw1974

No need to scream. Im monitoring every help channel every day and if people lose wdPost its most likely due to hw failure.
I will convert this back to a issue, I will also edit some try to make it clear what you want - please edit/share your correct lotus version lotus version.

[seantw1974]

Daemon: 1.15.1-dev+mainnet+git.2e22781f8.dirty+api1.3.0
Local: lotus-miner version 1.15.1-dev+mainnet+git.2e22781f8.dirty

I agrre it's "hw failure"
but this "hw failure" only make several sectors have issue .
and these issue can be found by command "lotus-miner proving check --slow 0"
but this way cannot resolve "one sector fail whole partition"
"Rust unwind panic will make windowpost stop to go on" that's why make "whole partion fail"
This is what should be fix !
Thanks for your help!!

[Reiers]

#8286

Your issue is back.

[Reiers]

#7971

This should solve the issue - just have to wait for it to land. @seantw1974

[LexLuthr]

I am archiving this discussion as it has been moved back to an issue.