Help, fault sectors by window post failed

[potato1992]

I lost 20% of my power, fault sectors by window post failed, all newly sealed sectors will be faulty.

The log:

2022-01-11T07:11:03.820+0800	ERROR	storageminer	storage/wdpost_run.go:49	Got err running window post failed:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
        /home/deci/Downloads/lotus_build/lotus_mainnet/storage/wdpost_run.go:693
  - generate_window_post: merkle_tree failed: SectorId(9107)
    
    Caused by:
        0: failed to read 32 bytes from file at offset 4908534016
        1: failed to fill whole buffer
    github.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt
    	/home/deci/Downloads/lotus_build/lotus_mainnet/extern/filecoin-ffi/proofs.go:684
    github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt
    	/home/deci/Downloads/lotus_build/lotus_mainnet/extern/sector-storage/ffiwrapper/verifier_cgo.go:46
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
    	/home/deci/Downloads/lotus_build/lotus_mainnet/storage/wdpost_run.go:639
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runGeneratePoST
    	/home/deci/Downloads/lotus_build/lotus_mainnet/storage/wdpost_run.go:103
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).startGeneratePoST.func1
    	/home/deci/Downloads/lotus_build/lotus_mainnet/storage/wdpost_run.go:87
    runtime.goexit
    	/usr/local/go/src/runtime/asm_amd64.s:1371 - TODO handle errors
2022-01-11T07:11:03.820+0800	WARN	storageminer	storage/wdpost_changehandler.go:255	Aborted window post Proving (Deadline: &{CurrentEpoch:1451662 PeriodStart:1451431 Index:3 Open:1451611 Close:1451671 Challenge:1451591 FaultCutoff:1451541 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})

It keeps trying this sector, and all failed, thus all later sector would not be pushed for Windowpost. Thus I have the fixed number of valid sectors everyday.
When this happened on the first day, I terminated the sectors. However, on the second day it showed again with a different sector id. I terminated it again and the same thing happens on the third day.

• Other information:
  The storage is a mounted samba folder, would this be the reason? I tried writing code to read the sector file at offset 4908534016 with length of 32, it succeed.

I have been fined for faulty for so many days, please help.

[SgtCoin]

Hi @potato1992

Lets see here,

If you run lotus-miner proving deadlines, you can check which deadline has the sector in question. Once you know the sector, you can run lotus-miner proving check --slow <deadline id> to see if the miner can check those sectors. If it still fails, delete the sector in question.

A few things to check.

• Make sure the connection is good between miner and storage
• Make sure that your user has the right read/write permissions.

Does the sector in question exist at all in your long term storage?

Did the Terminate message for the sector go out?

Disclaimer, this is NOT lotus software, but this can rebuild the sector if broken:
https://github.com/froghub-io/filecoin-sealer-recover

[potato1992]

IT SHOULD WORK! THANKS SO MUCH! Permission/connection are good, I found many bad sectors through the lotus-miner proving check --slow <deadline id> command, they come from the same raid group. I think I need check the disk.

[SgtCoin]

Hi @potato1992,

I am glad to hear you have found the suspected bad sectors. Please let us know if you need any help with the disks.

[potato1992]

Hello @SgtCoin , I setup a new raid 6 with 12 newly purchased disk, and moves the data(whole directory) from the old raid to the new raid, the problem magically comes to the new raid!

Newly sealed sectors would randomly come to this problem again. I have 3 other raids groups, they all works well....

[SgtCoin]

@potato1992

Are the other three raid groups mounted to the same location? I just want to figure out if samba is part of the issue.

[potato1992]

I have 3 machines.
Machine 1: running lotus miner and lotus, 2 raid groups, the 2 raid groups are all good.
Machine 2: samba mounted to machine 1, 2 raid groups, one raid is good but the other would cause the problem.
Now I have added Machine 3:
samba mounted to machine 1, one raid, I have moved the folder from the bad raid from machine 2, the old sectors are just fine, but some of the new sectors randomly go bad.

The SMART info for all disk are good, no error.
mdadm --details shows for all raild groups (12 drives raid6).

/dev/md127:
           Version : 1.2
     Creation Time : Tue Jan 25 12:40:05 2022
        Raid Level : raid6
        Array Size : 156257459200 (149018.73 GiB 160007.64 GB)
     Used Dev Size : 15625745920 (14901.87 GiB 16000.76 GB)
      Raid Devices : 12
     Total Devices : 12
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Wed Jan 26 14:33:55 2022
             State : active, resyncing 
    Active Devices : 12
   Working Devices : 12
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

     Resync Status : 84% complete

              Name : ubuntu9:0  (local to host ubuntu9)
              UUID : 7bb0ce1f:d156fa02:dc91dcb9:963e85ac
            Events : 37055

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1
       4       8       65        4      active sync   /dev/sde1
       5       8       81        5      active sync   /dev/sdf1
       6       8       97        6      active sync   /dev/sdg1
       7       8      113        7      active sync   /dev/sdh1
       8       8      145        8      active sync   /dev/sdj1
       9       8      161        9      active sync   /dev/sdk1
      10       8      177       10      active sync   /dev/sdl1
      11       8      129       11      active sync   /dev/sdi1

[SgtCoin]

@potato1992

Thank you for the information. What are the specs of the 3 hosts you have?

[potato1992]

@SgtCoin
machine1(miner+lotus):

AMD EPYC 7402 24-Core Processor
256G ecc DDR4 ram
2T os SSD+2T lotus data SSD
10G Ethernet

machine2/3(storage):

AMD EPYC 7232P 8-Core Processor
16G ecc DDR4 ram
512G os SSD
10G Ethernet

[potato1992]

@SgtCoin Thanks for your kind help. It has been solved. It turns out to be the issue of Samba. The problem is, samba cannot open more than (16384+40) files no matter how I adjust the ulimit (hard limit in code). The problem will arise when sealed sectors transmitted to storage machine and Windowed post are happening at the same time, Windowed post will use up the file limitation, then the samba transmitted file is broken.....

The solution can be two ways:

1. Modify the source code of samba, the following line:

#define MIN_OPEN_FILES_WINDOWS 16384

2. . Another solution is to switch into NFS, which support up to 32768 files, this is enough for lotus miner Windowed post(2349 sectors) and samba transmitting at the same time.