One sector already inside dealline but cannot do anything to remove ! Please help !

[seantw1974]

One Sector Alredy in Deadline (Partition 0),

CCIDcommD and CIDcommR: all are NIL？？？
Cannot Remove. Cannot Terminate, Cannot pass widnow Post....
Every Day have to pay Fine...
Please tell me how to resolve ?

root@miner-ip03:~/nvtop/build# lotus-miner sectors status --log 68
SectorID: 68
Status: PreCommit1
CIDcommD:
CIDcommR:
Ticket:
TicketH: 0
Seed:
SeedH: 0
Precommit:
Commit:
Deals: [0]
Retries: 0

Event Log:
0. 2022-02-03 13:30:10 +0800 CST: [event;sealing.SectorTerminate] {"User":{}}

[SgtCoin]

@seantw1974

Hi,

Is that only log entry for that sector? Is this a CC sector? When I look at a sector that was terminated, this (sealing.SectorTerminate) is the 18th entry, where the 0 entry is sealing.SectorStartCC.

If this is a CC sector, have you tried changing the state of the sector with lotus-miner sectors update-state?

[seantw1974]

Even we changed the status ...the data online still exist.. This sector still appear in "Deadline Sectors list"

[seantw1974]

We cannot find anyway to remove from "Deadline Sectors list"

[seantw1974]

miner: f01703785
sectors: 68 69 70

Already 7days cannot pass/cannot del/cannot cancel/cannot do anything !!!!!!! but always have to pay fine!!!!! What can do now ?????

root@miner-ip03:/nvme0# lotus-miner sectors status --log 68
SectorID: 68
Status: TerminateWait
CIDcommD:
CIDcommR:
Ticket:
TicketH: 0
Seed:
SeedH: 0
Precommit:
Commit:
Deals: [0]
Retries: 0

Event Log:
0. 2022-02-03 13:30:10 +0800 CST: [event;sealing.SectorTerminate] {"User":{}}

1. 2022-03-11 17:50:42 +0800 CST: [event;sealing.SectorForceState] {"User":{"State":"Packing"}}
2. 2022-03-11 17:50:42 +0800 CST: [event;sealing.SectorPacked] {"User":{"FillerPieces":[{"Size":2048,"PieceCID":{"/":"baga6ea4seaqpy7usqklokfx2vxuynmupslkeutzexe2uqurdg5vhtebhxqmpqmy"}}]}}
3. 2022-03-11 17:50:42 +0800 CST: [event;sealing.SectorRemove] {"User":{}}
4. 2022-03-11 17:50:42 +0800 CST: [event;sealing.SectorRemoved] {"User":{}}
5. 2022-03-11 18:48:51 +0800 CST: [event;sealing.SectorForceState] {"User":{"State":"Packing"}}
6. 2022-03-11 18:48:51 +0800 CST: [event;sealing.SectorPacked] {"User":{"FillerPieces":null}}
7. 2022-03-11 18:48:51 +0800 CST: [event;sealing.SectorRemove] {"User":{}}
8. 2022-03-11 18:48:51 +0800 CST: [event;sealing.SectorRemoved] {"User":{}}
9. 2022-03-11 19:06:34 +0800 CST: [event;sealing.SectorRemove] {"User":{}}
10. 2022-03-11 19:06:34 +0800 CST: [event;sealing.SectorRemoved] {"User":{}}
11. 2022-03-11 20:55:35 +0800 CST: [event;sealing.SectorTerminate] {"User":{}}
12. 2022-03-11 20:55:35 +0800 CST: [event;sealing.SectorTerminating] {"User":{"Message":null}}
13. 2022-03-11 21:58:47 +0800 CST: [event;sealing.SectorTerminate] {"User":{}}
14. 2022-03-11 21:58:47 +0800 CST: [event;sealing.SectorTerminating] {"User":{"Message":null}}
15. 2022-03-11 21:59:53 +0800 CST: [event;sealing.SectorRemove] {"User":{}}
16. 2022-03-11 21:59:53 +0800 CST: [event;sealing.SectorRemoved] {"User":{}}
17. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorForceState] {"User":{"State":"AddPiece"}}
18. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorPieceAdded] {"User":{"NewPieces":null}}
19. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorStartPacking] {"User":{}}
20. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorPacked] {"User":{"FillerPieces":null}}
21. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorRemove] {"User":{}}
22. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorRemoved] {"User":{}}
23. 2022-03-12 13:31:52 +0800 CST: [event;sealing.SectorForceState] {"User":{"State":"PreCommitFailed"}}
24. 2022-03-12 13:31:52 +0800 CST: [event;sealing.SectorSealPreCommit1Failed] {"User":{}}
    bad CommD error: on chain CommD differs from sector: baga6ea4seaqpy7usqklokfx2vxuynmupslkeutzexe2uqurdg5vhtebhxqmpqmy !=
25. 2022-03-12 13:32:52 +0800 CST: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}
26. 2022-03-19 19:44:12 +0800 CST: [event;sealing.SectorTerminate] {"User":{}}
27. 2022-03-19 19:44:12 +0800 CST: [event;sealing.SectorTerminating] {"User":{"Message":null}}

[seantw1974]

Please help to remove these 3 sectors from deadline sectors list !!

[SgtCoin]

@seantw1974

1. What is the output of lotus-miner sectors status --on-chain-info --log 68? I want to see what the chain thinks about this sector.
2. What is the output of lotus-miner storage find 68? I want to see where this currently resides and if it is present anywhere at all.

[TippyFlitsUK]

Hi @seantw1974

Let's do a bit more digging!!

Can you please provide the information that @SgtCoin has requested in the post above this one ^^.

It would also be very useful to get a better overview of your miner. Please run the command below and either post or link to the generated f01703785.log file to this thread.

lotus-miner info all > f01703785.log

Many thanks!!

[seantw1974]

`root@miner-ip03:~# lotus-miner sectors status --on-chain-info --log 68
SectorID: 68
Status: Removed
CIDcommD:
CIDcommR:
Ticket:
TicketH: 0
Seed:
SeedH: 0
Precommit:
Commit:
Deals: [0]
Retries: 0

Sector On Chain Info
SealProof: 8
Activation: 1515042
Expiration: 3064145
DealWeight: 0
VerifiedDealWeight: 0
InitialPledge: 149901057763566399

Expiration Info
OnTime: 0
Early: 1738717

Event Log:
0. 2022-02-03 13:30:10 +0800 CST: [event;sealing.SectorTerminate] {"User":{}}

1. 2022-03-11 17:50:42 +0800 CST: [event;sealing.SectorForceState] {"User":{"State":"Packing"}}
2. 2022-03-11 17:50:42 +0800 CST: [event;sealing.SectorPacked] {"User":{"FillerPieces":[{"Size":2048,"PieceCID":{"/":"baga6ea4seaqpy7usqklokfx2vxuynmupslkeutzexe2uqurdg5vhtebhxqmpqmy"}}]}}
3. 2022-03-11 17:50:42 +0800 CST: [event;sealing.SectorRemove] {"User":{}}
4. 2022-03-11 17:50:42 +0800 CST: [event;sealing.SectorRemoved] {"User":{}}
5. 2022-03-11 18:48:51 +0800 CST: [event;sealing.SectorForceState] {"User":{"State":"Packing"}}
6. 2022-03-11 18:48:51 +0800 CST: [event;sealing.SectorPacked] {"User":{"FillerPieces":null}}
7. 2022-03-11 18:48:51 +0800 CST: [event;sealing.SectorRemove] {"User":{}}
8. 2022-03-11 18:48:51 +0800 CST: [event;sealing.SectorRemoved] {"User":{}}
9. 2022-03-11 19:06:34 +0800 CST: [event;sealing.SectorRemove] {"User":{}}
10. 2022-03-11 19:06:34 +0800 CST: [event;sealing.SectorRemoved] {"User":{}}
11. 2022-03-11 20:55:35 +0800 CST: [event;sealing.SectorTerminate] {"User":{}}
12. 2022-03-11 20:55:35 +0800 CST: [event;sealing.SectorTerminating] {"User":{"Message":null}}
13. 2022-03-11 21:58:47 +0800 CST: [event;sealing.SectorTerminate] {"User":{}}
14. 2022-03-11 21:58:47 +0800 CST: [event;sealing.SectorTerminating] {"User":{"Message":null}}
15. 2022-03-11 21:59:53 +0800 CST: [event;sealing.SectorRemove] {"User":{}}
16. 2022-03-11 21:59:53 +0800 CST: [event;sealing.SectorRemoved] {"User":{}}
17. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorForceState] {"User":{"State":"AddPiece"}}
18. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorPieceAdded] {"User":{"NewPieces":null}}
19. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorStartPacking] {"User":{}}
20. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorPacked] {"User":{"FillerPieces":null}}
21. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorRemove] {"User":{}}
22. 2022-03-11 22:10:32 +0800 CST: [event;sealing.SectorRemoved] {"User":{}}
23. 2022-03-12 13:31:52 +0800 CST: [event;sealing.SectorForceState] {"User":{"State":"PreCommitFailed"}}
24. 2022-03-12 13:31:52 +0800 CST: [event;sealing.SectorSealPreCommit1Failed] {"User":{}}
    bad CommD error: on chain CommD differs from sector: baga6ea4seaqpy7usqklokfx2vxuynmupslkeutzexe2uqurdg5vhtebhxqmpqmy !=
25. 2022-03-12 13:32:52 +0800 CST: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}
26. 2022-03-19 19:44:12 +0800 CST: [event;sealing.SectorTerminate] {"User":{}}
27. 2022-03-19 19:44:12 +0800 CST: [event;sealing.SectorTerminating] {"User":{"Message":null}}
28. 2022-03-21 14:46:50 +0800 CST: [event;sealing.SectorForceState] {"User":{"State":"Removed"}}
    `

[seantw1974]

lotus-miner storage find 68
Already empty
if not remove , will cause whole partitioin error !!

[seantw1974]

Please help to check

http://61.216.11.93/f01703785.log

[TippyFlitsUK]

Hi @seantw1974

Thank you for posting your log file! I have examined the data in detail and would appreciate your clarification.

With regards to Window Post - You mentioned that your whole partition is failing. Looking at your lotus-miner proving deadlines and your most recent SubmitWindowedPoSt's for deadline 0 (listed below), it would appear that your recent window posts have actually been successful and 1247 sectors are proving and active on chain with the 3 faults you previously mentioned. Can you please confirm if you are seeing something different? Is your miner reporting window post failure??

#: Proving Deadlines
Miner: f01703785
deadline  partitions  sectors (faults)  proven partitions
0         1           1247 (3)          0
1         1           2 (0)             0
2         1           66 (0)            0

https://filfox.info/en/message/bafy2bzacecwnzlxrxdjwzas3omhtmu373ejgw4xlnaabt3zehhq2mtu745yca
https://filfox.info/en/message/bafy2bzacec5evm2zlxxs423um7pscknx7jcu6yc5dslz6ofldka3zr37lde7c
https://filfox.info/en/message/bafy2bzaceab3xuzh5mak7bq3jl2rmtrnnzevmmpzmirintzgn63r4ppzbobom

Can you also please advise when you last restarted your lotus-miner? (approximately)

Many thanks!!

[TippyFlitsUK]

Thank you. Understood!!

[seantw1974]

window post is 100% success now !! but these 3 sectors always in "Fail" status !!!
MUST pay fine!!!!

[TippyFlitsUK]

Thank you for clarifying @seantw1974 . How long ago (approximately) did you last restart your miner?

[seantw1974]

Around 48-72 hours

[TippyFlitsUK]

Thank you!!

[TippyFlitsUK]

Hi @seantw1974

I can see from your sector log that you have tried to terminate these sectors multiple times but looking at your miner transaction history I can also see that these terminate messages never actually arrived on chain. We need to make sure that these terminate messages are successfully submitted by Lotus. There are a couple of things that we can try, please run the following commands on your miner:

• touch /filstore/storage/sealed/s-t01703785-68
• Restart lotus-miner
• lotus-miner sectors remove --really-do-it 68
• lotus-miner sectors terminate --really-do-it 68

[seantw1974]

this already done before !! not working ...
due to my 68/69/70 's CIDcommD and CIDcommR all empty
All status are empty (Ticket/TicketH/Seed/SeedH/Precommit/Commit/Deals) !!!
Terminate and Remove cannot find anything to do !

SectorID: 68
Status: Removed
CIDcommD:
CIDcommR:
Ticket:
TicketH: 0
Seed:
SeedH: 0
Precommit:
Commit:
Deals: [0]
Retries: 0

[TippyFlitsUK]

Have you tried terminating through lotus-shed?

[seantw1974]

@TippyFlitsUK
Really thanks~
This way working finally~~
use lotus-shed can remove online data!!!!

[TippyFlitsUK]

That's great news @seantw1974 !! I will mark this as answered but please eel free to reply if you have any further issues!!

[TippyFlitsUK]

Resolved using lotus-shed terminate function.