Cannot remove sectors stuck in Committing #6522

jennijuju · 2020-09-29T04:05:48Z

jennijuju
Sep 29, 2020
Maintainer

After 0.8.0 upgrade, many miners reported that their sectors are stuck in Committing status（for days), and when they try to remove it using lotus-miner sectors remove, it doesnt work.

So I tried to reproduce the issue, and the remove does work after a while when the following shows in the log

0-09-28T23:55:43.296-0400	ERROR	evtsm	[email protected]/machine.go:83	Executing event planner failed: running planner for state Removing failed:
  github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).plan
    /Users/jennijuju/filecoin/software/lotus/extern/storage-sealing/fsm.go:205
 - planner for state Removing received unexpected event sealing.SectorCommitted ({User:{Proof:[175 221 6 162 182 25 128 3 115 35 185 16 162 68 101 206 189 78 215 111 209 138 170 171 252 214 190 160 212 57 16 212 64 41 9 91 73 60 200 177 246 89 34 3 209 227 66 143 181 65 35 219 168 142 68 9 80 18 165 208 49 58 200 231 236 117 134 240 60 69 239 255 4 71 132 72 162 128 214 25 234 66 220 86 76 86 96 171 34 246 180 63 23 199 198 19 21 97 124 132 72 133 98 171 107 4 116 109 37 105 208 135 68 220 96 197 70 194 212 52 110 123 216 43 101 60 233 209 166 40 177 71 188 249 232 245 31 124 129 130 139 47 102 224 179 47 239 72 107 157 64 231 145 241 206 20 150 68 105 110 36 178 143 198 245 228 219 148 152 231 235 100 213 129 241 93 93 91 17 97 135 112 86 153 218 172 227 146 181 74 214 134]}}):
  github.com/filecoin-project/lotus/extern/storage-sealing.planOne.func1

And the status of the sector is moved to Removing(and stays in Removing)

However, if we check the log of the sector, it seems like it is removed after Committing was successful and this is may be why the miner cannot remove it(since they arent able to finish committing)

0.	2020-09-28 23:53:25 -0400 EDT:	[event;sealing.SectorStartCC]	{"User":{"ID":5,"SectorType":0,"Pieces":[{"Piece":{"Size":2048,"PieceCID":{"/":"baga6ea4seaqpy7usqklokfx2vxuynmupslkeutzexe2uqurdg5vhtebhxqmpqmy"}},"DealInfo":null}]}}
1.	2020-09-28 23:53:25 -0400 EDT:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":null}}
2.	2020-09-28 23:53:25 -0400 EDT:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzJLaUJWMSIsImxhYmVscyI6eyJTdGFja2VkRHJnMktpQlYxIjp7ImxhYmVscyI6W3sicGF0aCI6Ii9Vc2Vycy9qZW5uaWp1anUvLmxvdHVzbWluZXIvY2FjaGUvcy10MDEwMDAtNSIsImlkIjoibGF5ZXItMSIsInNpemUiOjY0LCJyb3dzX3RvX2Rpc2NhcmQiOjV9LHsicGF0aCI6Ii9Vc2Vycy9qZW5uaWp1anUvLmxvdHVzbWluZXIvY2FjaGUvcy10MDEwMDAtNSIsImlkIjoibGF5ZXItMiIsInNpemUiOjY0LCJyb3dzX3RvX2Rpc2NhcmQiOjV9XSwiX2giOm51bGx9fSwiY29uZmlnIjp7InBhdGgiOiIvVXNlcnMvamVubmlqdWp1Ly5sb3R1c21pbmVyL2NhY2hlL3MtdDAxMDAwLTUiLCJpZCI6InRyZWUtZCIsInNpemUiOjEyNywicm93c190b19kaXNjYXJkIjo1fSwiY29tbV9kIjpbMjUyLDEyNiwxNDYsMTMwLDE1MCwyMjksMjIsMjUwLDE3MywyMzMsMTM0LDE3OCwxNDMsMTQ2LDIxMiw3NCw3OSwzNiwxODUsNTMsNzIsODIsMzUsNTUsMTA2LDEyMSwxNDQsMzksMTg4LDI0LDI0OCw1MV19","TicketValue":"tCcLpyM5cAV7DzafldVFyzw71SXxiNeGeDJMsWEV0j4=","TicketEpoch":1625}}
3.	2020-09-28 23:53:25 -0400 EDT:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcb2gxnutvshaopuklubxvifzut7nds6jvlw6ydkpxtcsrgixnatsi"},"Unsealed":{"/":"baga6ea4seaqpy7usqklokfx2vxuynmupslkeutzexe2uqurdg5vhtebhxqmpqmy"}}}
4.	2020-09-28 23:53:25 -0400 EDT:	[event;sealing.SectorPreCommitted]	{"User":{"Message":{"/":"bafy2bzaced43gwwx67r2q5k3kgaoijwbxurieosi6bozojoghcimmsvpz4qcu"},"PreCommitDeposit":"139371709306","PreCommitInfo":{"SealProof":0,"SectorNumber":5,"SealedCID":{"/":"bagboea4b5abcb2gxnutvshaopuklubxvifzut7nds6jvlw6ydkpxtcsrgixnatsi"},"SealRandEpoch":1625,"DealIDs":[],"Expiration":1552977,"ReplaceCapacity":false,"ReplaceSectorDeadline":0,"ReplaceSectorPartition":0,"ReplaceSectorNumber":0}}}
5.	2020-09-28 23:53:55 -0400 EDT:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIgcWjiSQ1f2zusMxJROicG0k5Bx3mcSzp5I/q4eGoUplo="}}
6.	2020-09-28 23:54:35 -0400 EDT:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"D4Sir42LqinvTclD9Mj9I4tl7yHY4L95bhOv+Kl48Sw=","SeedEpoch":2537}}
7.	2020-09-28 23:55:43 -0400 EDT:	[event;sealing.SectorRemove]	{"User":{}}
8.	2020-09-28 23:55:43 -0400 EDT:	[event;sealing.SectorCommitted]	{"User":{"Proof":"r90GorYZgANzI7kQokRlzr1O12/Riqqr/Na+oNQ5ENRAKQlbSTzIsfZZIgPR40KPtUEj26iORAlQEqXQMTrI5+x1hvA8Re//BEeESKKA1hnqQtxWTFZgqyL2tD8Xx8YTFWF8hEiFYqtrBHRtJWnQh0TcYMVGwtQ0bnvYK2U86dGmKLFHvPno9R98gYKLL2bgsy/vSGudQOeR8c4UlkRpbiSyj8b15NuUmOfrZNWB8V1dWxFhh3BWmdqs45K1StaG"}}
9.	2020-09-28 23:56:46 -0400 EDT:	[event;sealing.SectorRestart]	{"User":{}}
10.	2020-09-28 23:56:46 -0400 EDT:	[event;sealing.SectorRemoved]	{"User":{}}

So the question/issue is:

Why the sectors are stuck in committing?
Should committing have a timeout/retry?
How to properly remove them?

Answered by dgreuel2002

Feb 25, 2021

I did find a work around for this issue if anyone needs assistance in this.
After some research I found that the miner was running C1 on these sectors and it was then getting stuck looking for the data to move forward to the next step.
Mine was for sector 630

I then shut down my miner and made a back up.
I created a dummy file in the sealed/ folder of my sealing repo called s-t047419-630 (touch /sealingrepo/sealed/s-t047419-630)
I also created a empty dir in the /cache folder of my sealing repo called s-t047419-630 (mkdir /sealingrepo/cache/s-t047419-630)

After that I rebooted miner and it then found the bad files and removed the sector.
Here is the log after miner rebooted and finally re…

View full answer

jennijuju · 2020-09-29T04:06:55Z

jennijuju
Sep 29, 2020
Maintainer Author

dump from slack

hello,guys  there are some sectors's status is Committing which continued for 2 days, who could tell me how solve it?
[11:10 AM] 2765: Committing sSet: NO active: NO tktH: 58159 seedH: 61861 deals: [0]
3464: Committing sSet: NO active: NO tktH: 69953 seedH: 73997 deals: [0]
3465: Committing sSet: NO active: NO tktH: 69764 seedH: 73656 deals: [0]
3466: Committing sSet: NO active: NO tktH: 69764 seedH: 73889 deals: [0]
3470: Committing sSet: NO active: NO tktH: 69953 seedH: 73973 deals: [0]
3691: Committing sSet: NO active: NO tktH: 73447 seedH: 76090 deals: [0]
3693: Committing sSet: NO active: NO tktH: 73447 seedH: 76190 deals: [0]
4222: Committing sSet: NO active: NO tktH: 83115 seedH: 86739 deals: [0]
4574: Committing sSet: NO active: NO tktH: 83115 seedH: 86985 deals: [0]
4748: Committing sSet: NO active: NO tktH: 83281 seedH: 86726 deals: [0]
[11:10 AM] Event Log:
0.	2020-09-14 09:57:33 +0800 CST:	[event;sealing.SectorStartCC]	{"User":{"ID":2765,"SectorType":3,"Pieces":[{"Piece":{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}},"DealInfo":null}]}}
1.	2020-09-14 09:57:33 +0800 CST:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":null}}
2.	2020-09-14 18:08:52 +0800 CST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: handler: websocket connection closed
3.	2020-09-14 18:09:52 +0800 CST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
4.	2020-09-15 05:33:53 +0800 CST:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjEiLCJsYWJlbHMiOnsiU3RhY2tlZERyZzMyR2lCVjEiOnsibGFiZWxzIjpbeyJwYXRoIjoiL2xvdHVzX2hkL3dvcmtlci9jYWNoZS9zLXQwODA5NC0yNzY1IiwiaWQiOiJsYXllci0xIiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvbG90dXNfaGQvd29ya2VyL2NhY2hlL3MtdDA4MDk0LTI3NjUiLCJpZCI6ImxheWVyLTIiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9sb3R1c19oZC93b3JrZXIvY2FjaGUvcy10MDgwOTQtMjc2NSIsImlkIjoibGF5ZXItMyIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL2xvdHVzX2hkL3dvcmtlci9jYWNoZS9zLXQwODA5NC0yNzY1IiwiaWQiOiJsYXllci00Iiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvbG90dXNfaGQvd29ya2VyL2NhY2hlL3MtdDA4MDk0LTI3NjUiLCJpZCI6ImxheWVyLTUiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9sb3R1c19oZC93b3JrZXIvY2FjaGUvcy10MDgwOTQtMjc2NSIsImlkIjoibGF5ZXItNiIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL2xvdHVzX2hkL3dvcmtlci9jYWNoZS9zLXQwODA5NC0yNzY1IiwiaWQiOiJsYXllci03Iiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvbG90dXNfaGQvd29ya2VyL2NhY2hlL3MtdDA4MDk0LTI3NjUiLCJpZCI6ImxheWVyLTgiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9sb3R1c19oZC93b3JrZXIvY2FjaGUvcy10MDgwOTQtMjc2NSIsImlkIjoibGF5ZXItOSIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL2xvdHVzX2hkL3dvcmtlci9jYWNoZS9zLXQwODA5NC0yNzY1IiwiaWQiOiJsYXllci0xMCIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL2xvdHVzX2hkL3dvcmtlci9jYWNoZS9zLXQwODA5NC0yNzY1IiwiaWQiOiJsYXllci0xMSIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N31dLCJfaCI6bnVsbH19LCJjb25maWciOnsicGF0aCI6Ii9sb3R1c19oZC93b3JrZXIvY2FjaGUvcy10MDgwOTQtMjc2NSIsImlkIjoidHJlZS1kIiwic2l6ZSI6MjE0NzQ4MzY0Nywicm93c190b19kaXNjYXJkIjo3fSwiY29tbV9kIjpbNywxMjYsOTUsMjIyLDUzLDE5NywxMCwxNDcsMywxNjUsODAsOSwyMjcsNzMsMTM4LDc4LDE5MCwyMjMsMjQzLDE1Niw2NiwxODMsMTYsMTgzLDQ4LDIxNiwyMzYsMTIyLDE5OSwxNzUsMTY2LDYyXX0=","TicketValue":"6nagLoFOrVqQ1F7fenUoF2frXmwkrft7zTIwJS7g+Kw=","TicketEpoch":58159}}
5.	2020-09-15 10:32:08 +0800 CST:	[event;sealing.SectorSealPreCommit2Failed]	{"User":{}}
	seal pre commit(2) failed: handler: websocket connection closed
6.	2020-09-15 10:33:08 +0800 CST:	[event;sealing.SectorRetrySealPreCommit2]	{"User":{}}
7.	2020-09-15 16:15:12 +0800 CST:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcblistuirzvcpxkuqbhbqjjvhvb5gdp3rwuytrm7vru4trysyitzy"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
8.	2020-09-15 16:15:12 +0800 CST:	[event;sealing.SectorPreCommitted]	{"User":{"Message":{"/":"bafy2bzacedkh2dcwhw5fing4ml7we2hul2qgmaam27lr6rcbro2qeecsyi2yu"},"PreCommitDeposit":"352508269382884258","PreCommitInfo":{"SealProof":3,"SectorNumber":2765,"SealedCID":{"/":"bagboea4b5abcblistuirzvcpxkuqbhbqjjvhvb5gdp3rwuytrm7vru4trysyitzy"},"SealRandEpoch":58159,"DealIDs":[],"Expiration":1615238,"ReplaceCapacity":false,"ReplaceSectorDeadline":0,"ReplaceSectorPartition":0,"ReplaceSectorNumber":0}}}
9.	2020-09-15 16:18:30 +0800 CST:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIgTCjjQtdg5jWp+PgPQKEzpmwco5JPfe3maCQ/dNIiwXQBcaDkAiCPaQ4YXGivxI23fJsOyAYsrRn0s/J0c//98MEAGVSLpgFxoOQCINIxTe7oPPETvmcW359Whll5fN7jePGtZmEsc50z/S9x"}}
10.	2020-09-15 17:33:30 +0800 CST:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"HJYoNHCgZkSmq3M4w+OhqP/bJiJnnFPMiRd6bZTh2gw=","SeedEpoch":61861}}
11.	2020-09-16 16:37:22 +0800 CST:	[event;sealing.SectorRestart]	{"User":{}}
12.	2020-09-26 11:47:27 +0800 CST:	[event;sealing.SectorRestart]	{"User":{}}
13.	2020-09-26 12:33:24 +0800 CST:	[event;sealing.SectorRestart]	{"User":{}}
14.	2020-09-26 17:32:24 +0800 CST:	[event;sealing.SectorRestart]	{"User":{}}
15.	2020-09-26 20:06:17 +0800 CST:	[event;sealing.SectorRestart]	{"User":{}}
16.	2020-09-27 12:16:43 +0800 CST:	[event;sealing.SectorRestart]	{"User":{}}

@magik6k can a proof generated by the seed before upgrade still be committed after the upgrade epoch for this case?

0 replies

bobjiang · 2020-09-29T05:53:32Z

bobjiang
Sep 29, 2020

Here are my issues about faulty sectors:

e.g I have 2 sectors (40 & 41)

40: Committing      sSet: NO  active: NO  tktH: 34003 seedH: 36939 deals: [0]                                                                                                                       toUpgrade:false
41: FinalizeSector  sSet: YES active: NO  tktH: 40166 seedH: 44006 deals: [0]                                                                                                                       toUpgrade:false

I am trying to to remove them with 2 methods:
Method 1:

lotus-miner sectors remove --really-do-it 40
lotus-miner sectors remove --really-do-it 41

Method2:

lotus-miner sectors update-state --really-do-it 40 FailedUnrecoverable
lotus-miner sectors update-state --really-do-it 41 FailedUnrecoverable
==========================
after 10 hours
==========================
40: Committing      sSet: NO  active: NO  tktH: 34003 seedH: 36939  deals: [0]                                                                                                                       toUpgrade:false
41: FinalizeSector  sSet: YES active: NO  tktH: 40166 seedH: 44006  deals: [0]                                                                                                                       toUpgrade:false

And the sector status is following:

9.	2020-09-06 16:34:11 +0000 UTC:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcbdsj7yvkumt7ylksgkvcihlzvjdndxcdlajvfwr3t5bhfqq3nmkj"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
10.	2020-09-06 16:34:11 +0000 UTC:	[event;sealing.SectorPreCommitted]	{"User":{"Message":{"/":"bafy2bzaceac7jezx7ngroz3cua3asmca7kyreqj6uhwztkjl4zoxehwvk5ioq"},"PreCommitDeposit":"458713655715017378","PreCommitInfo":{"SealProof":3,"SectorNumber":40,"SealedCID":{"/":"bagboea4b5abcbdsj7yvkumt7ylksgkvcihlzvjdndxcdlajvfwr3t5bhfqq3nmkj"},"SealRandEpoch":34003,"DealIDs":[],"Expiration":1588111,"ReplaceCapacity":false,"ReplaceSectorDeadline":0,"ReplaceSectorPartition":0,"ReplaceSectorNumber":0}}}
11.	2020-09-06 16:37:30 +0000 UTC:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIgpRhPhGnBwnOrCtB1fAU1mYsCdjKWgzUkbrBfXr++cdgBcaDkAiCoRYy1yLj9eRbwFKmeLL/rokPuUU9WI6twlrr4H4T7IQFxoOQCIHkFh/hWAGYBpeEMw4b0ZLyisgVOz2bu0W6SPlvV8TsS"}}
12.	2020-09-06 17:52:31 +0000 UTC:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"Q69Lbevw8i3QsUWy28+Vwi6yAFCWewm5aJwa08XHTak=","SeedEpoch":36939}}
13.	2020-09-06 18:53:33 +0000 UTC:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): handler: websocket connection closed
14.	2020-09-06 18:54:33 +0000 UTC:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
15.	2020-09-07 06:53:17 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
16.	2020-09-08 04:15:05 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
17.	2020-09-10 02:07:30 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
18.	2020-09-10 02:40:36 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
19.	2020-09-11 23:19:13 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
20.	2020-09-12 05:18:52 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
21.	2020-09-14 00:14:01 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
22.	2020-09-17 05:52:31 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
23.	2020-09-17 06:51:17 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
24.	2020-09-17 22:59:15 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
25.	2020-09-21 22:41:20 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
26.	2020-09-26 11:31:38 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
27.	2020-09-26 12:33:29 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
28.	2020-09-27 04:07:24 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
29.	2020-09-28 00:37:52 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}



1504.	2020-09-09 03:21:36 +0000 UTC:	[event;sealing.SectorRetryPreCommit]	{"User":{}}
1505.	2020-09-09 03:21:36 +0000 UTC:	[event;sealing.SectorChainPreCommitFailed]	{"User":{}}
	pushing message to mpool: GasEstimateMessageGas error: estimating gas used: message execution failed: exit 19, reason: insufficient funds for pre-commit deposit: 442720272584365491 (RetCode=19)
1506.	2020-09-09 03:22:36 +0000 UTC:	[event;sealing.SectorRetryPreCommit]	{"User":{}}
1507.	2020-09-09 03:22:36 +0000 UTC:	[event;sealing.SectorPreCommitted]	{"User":{"Message":{"/":"bafy2bzacede3tfeeefhzxvkuqvyvzczvcffxdpaeizguz7elxtvdlih3pexrk"},"PreCommitDeposit":"488249512923751738","PreCommitInfo":{"SealProof":3,"SectorNumber":41,"SealedCID":{"/":"bagboea4b5abcbshvgybvgl3zddo6oahbiouwbrjddsmfqg7knjjdnwktuarxiubf"},"SealRandEpoch":40166,"DealIDs":[],"Expiration":1596751,"ReplaceCapacity":false,"ReplaceSectorDeadline":0,"ReplaceSectorPartition":0,"ReplaceSectorNumber":0}}}
1508.	2020-09-09 03:42:08 +0000 UTC:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIgLmPgHCszxYlf65a63K1OQuo/L5gTqXD3nScMc48QEiUBcaDkAiBaeApixqCiBPDTtI/CvVA8o8ZhPnFs5wQzv+oRiSKuCwFxoOQCIG4FzF0GjHZXaswunZYeg0oZNh8mDe1nBxT3nHYDMMh1"}}
1509.	2020-09-09 04:52:06 +0000 UTC:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"ZJa4zny9n46Ty9ibneGwv26hu68v0oAyMVylKm4w/Q8=","SeedEpoch":44006}}
1510.	2020-09-09 08:17:45 +0000 UTC:	[event;sealing.SectorCommitted]	{"User":{"Proof":"ofd+uMJ8eDjqa8jbv591BXxu6nl9BTA2gFebs+drlE3bf6gJnCH866mzVUxclPAol8rRaPHhyv4/in3Lb7fwMJ+0FMZV84Kc246p/X5BFaGVV+ytykls73XAyNtxtsKACWXN6bn/AyidrhUbXZ5Cq4uVpHuwYS/ZNR+2MCFp/ob957Uj2H+UzlKa6EL+TB0QtAIHknW98jetVAGLDmCq+OC5s8y9qMD1ByoeYDWkj4jGn5h3Y8BwQX+GaZFs1a+tq3eUNXzgn1IgO/wVjiOQDqEJX1uRgz3ZPMhT65e1EqQliKyEY4T2t//sK+/2ISjprkn8jf7YV1NfT+UZczz+dD9SSDgl3zsqYRcWhhrG9YKzDwV+Rz6h1AONmdPS53CXEccTJ1F9WH+1a72aFtxuoPaKhTKpWrm9/CVru4nnfSVXeGLFnq/mt/NHEmHTyUYrrpduhlwkoBPPjTHho6q143TEAFagXfYQwfmV9gD8yND7Hv3vv/bZaBkdPtthZSNSjG57e/BVfvz4QlRHG55e2apLDs6NqrxbSBLCDBSQCXInRIKCQKLdmpvpIrPq9sm+tfc8QdF28CDsHRFdhbAwusLbqnqRXlu5kgjKK1K7k2sDzndFRBOfJi+1mpazr3/cEYseyrEkPOLKkg7bNQ89R3cKrlLTQtI9inuVO1eMthJG+VrqyafQoOrvJAaSmSA3kyS2N5bFS+dJYTrjwhKsKgXCrcuKR/D9n8osRcBIyXjLs5biVcu6zxM81c8/oWdXtmSq+5LtwHKu25uVP/WXoAH6WC/ryYFkAlG1NeMRM1DI4kHx+TeX+DrT3lJmzXaWgCbff9feAngjnxVqOJ0zindueCwjLXCWJRgNhCN5Wx9Z5dAMEkKaqLILvIEqggTsAj6USS22m8MDAuFhRdfoEDJp1SANJVWyZOFJHShWeknbvsLt+ZvHgPVFHL9GvWqPtYECT8bY/rS8INS9zu6jlhR34qHHiq3rxiSFZXk3FVK8CJm6jD+UgbVsugH87lzOtEShcf6WH9T+Ba9Noew3DEMyBw7eIo7M3+c/8rgFBeWEkRphzNEVGDaf+irWn4Duhw+ytNzashJYd9RLOUTehYqEr6Q5ZD37RfQpFXLln+RfeXnfmESB/C5QPUfiOUJWBL4PMOhThCpZckwZujVUnikSBBxxMOPEd80oBpgkWn9eEMIiztvlx3aXGlMq5k8GqPma6WLp5HOpv8KXMqu1ir2CeHuuKlWLQv11ix0TAELhW1JKj10Ws9WAezAvzPzFovl2oQOKybL40NDxyX1+USU8WizOUx20HodFDJwADHO8SpPtOK+FFW+Sy9kcOM58qe6tLukGJHyXsbaZ3iIn3+pLF5GuZ9RoHxpkg8F+4AIWC57SPnMZWCqOKHbon/12CexR3KrfH6sVl6tFNHHMEzUPvQBOoWpQkMp/vr1U94Pblk3jExkQ3sxt79WHiPBOj1rlOtU3WmkePYqWE943/lKXuu/gRSu70CyWMolr39H6ohTmLTAw1SMJ8eEX1oy0tPs2L734mYa5DNaIsWZY7Bgc7lMFFIlpEklSAex0jMZW9WQZSO04gwLgxuFaI9xWgd70wy8ys/skTTOT7MAJKvooRgLY2eRb6X4rzFhu9pbmbe21rW/t33JRjDY8HPwiAsZMNkZToi31WVQzjhN/nx+TAFreXEK+McSM1dd51z1JTPsTtatSHtNaWfxmIceoj64Q3mqcULOa3L9+jbqAsrTxB8sn9a+R+FxA85h5mGG3QJ+VvZhHMc3hKogFiU9rl1BCykIZkDIHFoaCTUA1RasnVWiELCo4VD0CTsX6FRLpJA0lTBZbdB6x5/SiWkJYgV8d3yYBfJwKZbsG5tartJUXXMuay0D7CSub9pGY7XrsUErRCfZb33rJTfHhMiB3CxycQrcGLPXv0WdhWtloIggoSXXGdc3p1HCqAMh7sGwCG+7Zy1dEjaV3TKnP7DmalETjLyRNQq0dfhGp1JNeySgP36fgPHrORzIWYng1DllzQrB5bIMjyTjTbmZ26pyKgPoJGeW8lQlcHj1AbwM+xFQN9L/jDR2gZoihbowcX0XHzFmY+f+nHSztXs1SIoKcisnrlW6znnY77V6oQDSPdPhfc3bYY9pAjux+ogqFUUCRwBDuiCrOCJX1pvFVPH/VDNPO84QwNR8HLLvN8eXfP/OkJcn1KYdnsMrgJFkUB3QutViH8SmLjexblw19qxQ6l3zfDIF2MevRWu6dIZOSVGJ6MTxF/ibl5dyKJdXuzSWAO55xCsncBzSIgK04ADLhhgXitnFDM8Q3ulfQtMs2lytCEWq3AWH/JTPG7eQO6AEbW3uVKzoIougYzc4Y+Flwlze2+c9MZJn/qUC47C+b5dtI07tmDiagGuM4TFWJi7vXWsbZrfA1NIatefFxhphxBPYPcb6sksTQxmGIZ46L/S90z5VXGIm/shpDe16uY95R8bxFKuqevQIOxQLEUE+MuQOA+QxZ6tp9CGwOZ0eTjFunxaQXSiA8KGw/zZwmwoXWj4BkzEqDgyMsf8qi9DwF"}}
1511.	2020-09-09 08:17:45 +0000 UTC:	[event;sealing.SectorCommitSubmitted]	{"User":{"Message":{"/":"bafy2bzacedyepl7n3m3aqahro77lxituiri4my6j3fommnjz6zmcdxzzuagde"}}}
1512.	2020-09-09 08:51:00 +0000 UTC:	[event;sealing.SectorProving]	{"User":{}}
1513.	2020-09-10 02:07:30 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1514.	2020-09-10 02:40:36 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1515.	2020-09-11 23:19:13 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1516.	2020-09-12 05:18:52 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1517.	2020-09-14 00:14:01 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1518.	2020-09-17 05:52:31 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1519.	2020-09-17 06:51:17 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1520.	2020-09-17 22:59:15 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1521.	2020-09-21 22:41:20 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1522.	2020-09-26 11:31:38 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1523.	2020-09-26 12:33:29 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}
1524.	2020-09-27 04:07:24 +0000 UTC:	[event;sealing.SectorRestart]	{"User":{}}

0 replies

jennijuju · 2020-11-09T20:44:44Z

jennijuju
Nov 9, 2020
Maintainer Author

turns out sectors in commitwait cannot be removed either(Note: files for this sector are removed manually, too). seems like there is a pattern that sectors are stuck in SectorRestart.
chris_full_miner_info_all.txt

0 replies

jennijuju · 2020-11-11T22:22:59Z

jennijuju
Nov 11, 2020
Maintainer Author

It'd be great if we can get this looked at soon, since this is one of the top5 pain point for miners rn.

0 replies

askender · 2020-11-13T04:42:22Z

askender
Nov 13, 2020

ID State OnChain Active Expiration Deals DealWeight
20 FinalizeSector YES NO 771070 (in 26 weeks 5 days) 1 126.7KiB

2020-09-14 21:59:51 +0800 CST: [event;sealing.SectorCommitSubmitted] {"User":{"Message":{"/":"bafy2bzacecjqvj722xr6vveemhjguryxtci6buq5vd47mikdmf6ty5d4k4u7a"}}}
2020-09-14 22:01:31 +0800 CST: [event;sealing.SectorCommitFailed] {"User":{}}
failed to wait for porep inclusion: handler: websocket connection closed
2020-09-17 00:27:14 +0800 CST: [event;sealing.SectorForceState] {"User":{"State":"Committing"}}
2020-09-17 00:34:10 +0800 CST: [event;sealing.SectorRetryCommitWait] {"User":{}}
2020-09-17 00:37:48 +0800 CST: [event;sealing.SectorProving] {"User":{}}
2020-09-17 21:43:20 +0800 CST: [event;sealing.SectorRestart] {"User":{}}
2020-09-20 02:02:34 +0800 CST: [event;sealing.SectorRestart] {"User":{}}

lotus-miner sectors update-state --really-do-it 20 Removing
the command cannot remove it.

Doesnot know if it is related.

0 replies

dgreuel2002 · 2021-02-25T03:59:46Z

dgreuel2002
Feb 25, 2021

I did find a work around for this issue if anyone needs assistance in this.
After some research I found that the miner was running C1 on these sectors and it was then getting stuck looking for the data to move forward to the next step.
Mine was for sector 630

I then shut down my miner and made a back up.
I created a dummy file in the sealed/ folder of my sealing repo called s-t047419-630 (touch /sealingrepo/sealed/s-t047419-630)
I also created a empty dir in the /cache folder of my sealing repo called s-t047419-630 (mkdir /sealingrepo/cache/s-t047419-630)

After that I rebooted miner and it then found the bad files and removed the sector.
Here is the log after miner rebooted and finally removed the sector stuck in committing that was causing a never ending loop on the scheduler.

2021-02-11T15:45:36.922-0600 INFO stores stores/remote.go:334 Delete http://10.0.120.250:3456/remote/sealed/s-t047419-630
2021-02-11T15:45:36.924-0600 INFO stores stores/remote.go:334 Delete http://10.0.120.250:3456/remote/cache/s-t047419-630
2021-02-11T15:45:36.925-0600 ERROR evtsm [email protected]/machine.go:83 Executing event planner failed: running planner for state Removing failed:
github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).plan
/filecoin/build/lotus/extern/storage-sealing/fsm.go:233

planner for state Removing received unexpected event sealing.SectorComputeProofFailed ({User:computing seal proof failed(1): storage call error 0: StandaloneSealCommit: Replica /mnt/md0/worker1/sealed/s-t047419-630 exists, but is empty!}):
github.com/filecoin-project/lotus/extern/storage-sealing.planOne.func1
/filecoin/build/lotus/extern/storage-sealing/fsm.go:547

After this I then saw a state of removing and it was gone from the scheduler.
One more reboot and the sector was finally in a removed status.

0 replies

sectrgt · 2021-02-26T02:18:44Z

sectrgt
Feb 26, 2021

This solution presented above has worked for me after experiencing this exact issue

0 replies

RobQuistNL · 2021-10-21T01:42:41Z

RobQuistNL
Oct 21, 2021

Thats a fun workaround, but shouldn't we just be able to terminate sectors properly?

0 replies

Cannot remove sectors stuck in Committing #6522

Uh oh!

jennijuju Sep 29, 2020 Maintainer

Replies: 8 comments

Uh oh!

jennijuju Sep 29, 2020 Maintainer Author

Uh oh!

bobjiang Sep 29, 2020

Uh oh!

jennijuju Nov 9, 2020 Maintainer Author

Uh oh!

jennijuju Nov 11, 2020 Maintainer Author

Uh oh!

Uh oh!

askender Nov 13, 2020

Uh oh!

Uh oh!

dgreuel2002 Feb 25, 2021

Uh oh!

sectrgt Feb 26, 2021

Uh oh!

RobQuistNL Oct 21, 2021

jennijuju
Sep 29, 2020
Maintainer

jennijuju
Sep 29, 2020
Maintainer Author

bobjiang
Sep 29, 2020

jennijuju
Nov 9, 2020
Maintainer Author

jennijuju
Nov 11, 2020
Maintainer Author

askender
Nov 13, 2020

dgreuel2002
Feb 25, 2021

sectrgt
Feb 26, 2021

RobQuistNL
Oct 21, 2021