Unable to Recover Sectors Following Following Sealing Disruption #8395

iseriser · 2022-03-28T18:41:47Z

iseriser
Mar 28, 2022

Checklist

This is not a security-related bug/issue. If it is, please follow please follow the security policy.
This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
This is not a new feature request. If it is, please file a feature request instead.
This is not an enhancement request. If it is, please file a improvement suggestion instead.
I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
I did not make any code changes to lotus.

Lotus component

Lotus Version

Daemon:1.15.0+mainnet+git.0ac1bbc7a
Miner:1.15.0+mainnet+git.0ac1bbc7a

Describe the Bug

I have two sectors stuck with the following:

$lotus-miner sectors status --log --on-chain-info 356
SectorID:	356
Status:		Committing
CIDcommD:	baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
CIDcommR:	bagboea4b5abcai5jtzzubuvqai5a5aqodax4pjptejaihjupe2xbkpemucompqjr
Ticket:		c3047d579e825962fd1f796f5e3e45aff3025606d081d097ec7feb2f1194d40f
TicketH:	1664860
Seed:		6dd650f628fb4090bb1daa1d6c2d183781b06e736707c0293f224caed19d7736
SeedH:		1667003
Precommit:	bafy2bzacea4d3dkumfqaqgyuwvlmbrq7uyougynnpo6yewb7cclmxfmnr3ezu
Commit:		<nil>
Deals:		[0]
Retries:	0

Sector On Chain Info
SealProof:		8
Activation:		0
Expiration:		0
DealWeight:		0
VerifiedDealWeight:		0
InitialPledge:		0

Expiration Info
OnTime:		0
Early:		0
--------
Event Log:
0.	2022-03-26 07:18:01 +0000 UTC:	[event;sealing.SectorStartCC]	{"User":{"ID":356,"SectorType":8}}
1.	2022-03-26 07:20:13 +0000 UTC:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}]}}
2.	2022-03-26 07:20:13 +0000 UTC:	[event;sealing.SectorTicket]	{"User":{"TicketValue":"wwR9V56CWWL9H3lvXj5Fr/MCVgbQgdCX7H/rLxGU1A8=","TicketEpoch":1664860}}
3.	2022-03-26 11:20:24 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 101: worker restarted
4.	2022-03-26 11:21:24 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
5.	2022-03-26 14:55:10 +0000 UTC:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJ3d1I5VjU2Q1dXTDlIM2x2WGo1RnIvTUNWZ2JRZ2RDWDdIL3JMeEdVMUE4PSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvbG90dXMtc2VhbGluZy9zZWFsaW5nL2NhY2hlL3MtdDAxNzg0NDU4LTM1NiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MjE0NzQ4MzY0N30sImxhYmVscyI6eyJTdGFja2VkRHJnMzJHaUJWMSI6eyJfaCI6bnVsbCwibGFiZWxzIjpbeyJpZCI6ImxheWVyLTEiLCJwYXRoIjoiL2xvdHVzLXNlYWxpbmcvc2VhbGluZy9jYWNoZS9zLXQwMTc4NDQ1OC0zNTYiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci0yIiwicGF0aCI6Ii9sb3R1cy1zZWFsaW5nL3NlYWxpbmcvY2FjaGUvcy10MDE3ODQ0NTgtMzU2Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvbG90dXMtc2VhbGluZy9zZWFsaW5nL2NhY2hlL3MtdDAxNzg0NDU4LTM1NiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTQiLCJwYXRoIjoiL2xvdHVzLXNlYWxpbmcvc2VhbGluZy9jYWNoZS9zLXQwMTc4NDQ1OC0zNTYiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci01IiwicGF0aCI6Ii9sb3R1cy1zZWFsaW5nL3NlYWxpbmcvY2FjaGUvcy10MDE3ODQ0NTgtMzU2Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvbG90dXMtc2VhbGluZy9zZWFsaW5nL2NhY2hlL3MtdDAxNzg0NDU4LTM1NiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTciLCJwYXRoIjoiL2xvdHVzLXNlYWxpbmcvc2VhbGluZy9jYWNoZS9zLXQwMTc4NDQ1OC0zNTYiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci04IiwicGF0aCI6Ii9sb3R1cy1zZWFsaW5nL3NlYWxpbmcvY2FjaGUvcy10MDE3ODQ0NTgtMzU2Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvbG90dXMtc2VhbGluZy9zZWFsaW5nL2NhY2hlL3MtdDAxNzg0NDU4LTM1NiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTEwIiwicGF0aCI6Ii9sb3R1cy1zZWFsaW5nL3NlYWxpbmcvY2FjaGUvcy10MDE3ODQ0NTgtMzU2Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTEiLCJwYXRoIjoiL2xvdHVzLXNlYWxpbmcvc2VhbGluZy9jYWNoZS9zLXQwMTc4NDQ1OC0zNTYiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
6.	2022-03-26 15:25:11 +0000 UTC:	[event;sealing.SectorSealPreCommit2Failed]	{"User":{}}
	seal pre commit(2) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
7.	2022-03-26 15:26:11 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit2]	{"User":{}}
8.	2022-03-26 16:25:16 +0000 UTC:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcai5jtzzubuvqai5a5aqodax4pjptejaihjupe2xbkpemucompqjr"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
9.	2022-03-26 16:25:57 +0000 UTC:	[event;sealing.SectorPreCommitted]	{"User":{"Message":{"/":"bafy2bzacea4d3dkumfqaqgyuwvlmbrq7uyougynnpo6yewb7cclmxfmnr3ezu"},"PreCommitDeposit":"12691768088194580","PreCommitInfo":{"SealProof":8,"SectorNumber":356,"SealedCID":{"/":"bagboea4b5abcai5jtzzubuvqai5a5aqodax4pjptejaihjupe2xbkpemucompqjr"},"SealRandEpoch":1664860,"DealIDs":[],"Expiration":3216279,"ReplaceCapacity":false,"ReplaceSectorDeadline":0,"ReplaceSectorPartition":0,"ReplaceSectorNumber":0}}}
10.	2022-03-26 16:46:15 +0000 UTC:	[event;sealing.SectorChainPreCommitFailed]	{"User":{}}
	handler: websocket connection closed
11.	2022-03-27 09:35:46 +0000 UTC:	[event;sealing.SectorForceState]	{"User":{"State":"PreCommitting"}}
12.	2022-03-27 09:35:46 +0000 UTC:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIgKp0mXODNa1bUi3jyibLOFmX8INXcf1JXc/I6n+AhO88BcaDkAiCN+NJ+SYs4HirrwIC6xLf07tDOlzikdfR+DTdJ6jF8Zw=="}}
13.	2022-03-27 09:35:46 +0000 UTC:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"bdZQ9ij7QJC7HaodbC0YN4GwbnNnB8ApPyJMrtGddzY=","SeedEpoch":1667003}}

$lotus-miner sectors status --log --on-chain-info 357
SectorID:	357
Status:		Committing
CIDcommD:	baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
CIDcommR:	bagboea4b5abcarwu4iyj53puuzqalzzhiglx7s2welgkuudi3fs5lntawicguriu
Ticket:		e2f6eee68dd0a188c5e688adc524c3ac4ac10bacf6a7fe2589db9e065649d365
TicketH:	1664882
Seed:		35426435e45b012dc94310a0beec71c535c7d60732d2e54418b5da75becea615
SeedH:		1667036
Precommit:	bafy2bzacedekkjx75q4cwdjjdbzwktqn7pook525rftx5jjz2cg7xc3xl3i6g
Commit:		<nil>
Deals:		[0]
Retries:	0

Sector On Chain Info
SealProof:		8
Activation:		0
Expiration:		0
DealWeight:		0
VerifiedDealWeight:		0
InitialPledge:		0

Expiration Info
OnTime:		0
Early:		0
--------
Event Log:
0.	2022-03-26 07:30:01 +0000 UTC:	[event;sealing.SectorStartCC]	{"User":{"ID":357,"SectorType":8}}
1.	2022-03-26 07:31:29 +0000 UTC:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}]}}
2.	2022-03-26 07:31:29 +0000 UTC:	[event;sealing.SectorTicket]	{"User":{"TicketValue":"4vbu5o3QoYjF5oitxSTDrErBC6z2p/4lidueBlZJ02U=","TicketEpoch":1664882}}
3.	2022-03-26 11:19:24 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 101: worker restarted
4.	2022-03-26 11:20:24 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
5.	2022-03-26 15:31:06 +0000 UTC:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiI0dmJ1NW8zUW9ZakY1b2l0eFNURHJFckJDNnoycC80bGlkdWVCbFpKMDJVPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvbG90dXMtc2VhbGluZy9zZWFsaW5nL2NhY2hlL3MtdDAxNzg0NDU4LTM1NyIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MjE0NzQ4MzY0N30sImxhYmVscyI6eyJTdGFja2VkRHJnMzJHaUJWMSI6eyJfaCI6bnVsbCwibGFiZWxzIjpbeyJpZCI6ImxheWVyLTEiLCJwYXRoIjoiL2xvdHVzLXNlYWxpbmcvc2VhbGluZy9jYWNoZS9zLXQwMTc4NDQ1OC0zNTciLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci0yIiwicGF0aCI6Ii9sb3R1cy1zZWFsaW5nL3NlYWxpbmcvY2FjaGUvcy10MDE3ODQ0NTgtMzU3Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvbG90dXMtc2VhbGluZy9zZWFsaW5nL2NhY2hlL3MtdDAxNzg0NDU4LTM1NyIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTQiLCJwYXRoIjoiL2xvdHVzLXNlYWxpbmcvc2VhbGluZy9jYWNoZS9zLXQwMTc4NDQ1OC0zNTciLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci01IiwicGF0aCI6Ii9sb3R1cy1zZWFsaW5nL3NlYWxpbmcvY2FjaGUvcy10MDE3ODQ0NTgtMzU3Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvbG90dXMtc2VhbGluZy9zZWFsaW5nL2NhY2hlL3MtdDAxNzg0NDU4LTM1NyIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTciLCJwYXRoIjoiL2xvdHVzLXNlYWxpbmcvc2VhbGluZy9jYWNoZS9zLXQwMTc4NDQ1OC0zNTciLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci04IiwicGF0aCI6Ii9sb3R1cy1zZWFsaW5nL3NlYWxpbmcvY2FjaGUvcy10MDE3ODQ0NTgtMzU3Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvbG90dXMtc2VhbGluZy9zZWFsaW5nL2NhY2hlL3MtdDAxNzg0NDU4LTM1NyIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTEwIiwicGF0aCI6Ii9sb3R1cy1zZWFsaW5nL3NlYWxpbmcvY2FjaGUvcy10MDE3ODQ0NTgtMzU3Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTEiLCJwYXRoIjoiL2xvdHVzLXNlYWxpbmcvc2VhbGluZy9jYWNoZS9zLXQwMTc4NDQ1OC0zNTciLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
6.	2022-03-26 16:40:54 +0000 UTC:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcarwu4iyj53puuzqalzzhiglx7s2welgkuudi3fs5lntawicguriu"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
7.	2022-03-26 16:42:24 +0000 UTC:	[event;sealing.SectorPreCommitted]	{"User":{"Message":{"/":"bafy2bzacedekkjx75q4cwdjjdbzwktqn7pook525rftx5jjz2cg7xc3xl3i6g"},"PreCommitDeposit":"12691665787211640","PreCommitInfo":{"SealProof":8,"SectorNumber":357,"SealedCID":{"/":"bagboea4b5abcarwu4iyj53puuzqalzzhiglx7s2welgkuudi3fs5lntawicguriu"},"SealRandEpoch":1664882,"DealIDs":[],"Expiration":3216284,"ReplaceCapacity":false,"ReplaceSectorDeadline":0,"ReplaceSectorPartition":0,"ReplaceSectorNumber":0}}}
8.	2022-03-26 16:46:15 +0000 UTC:	[event;sealing.SectorChainPreCommitFailed]	{"User":{}}
	handler: websocket connection closed
9.	2022-03-27 09:35:52 +0000 UTC:	[event;sealing.SectorForceState]	{"User":{"State":"PreCommitting"}}
10.	2022-03-27 09:35:52 +0000 UTC:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIgKp0mXODNa1bUi3jyibLOFmX8INXcf1JXc/I6n+AhO88BcaDkAiCN+NJ+SYs4HirrwIC6xLf07tDOlzikdfR+DTdJ6jF8Zw=="}}
11.	2022-03-27 09:35:52 +0000 UTC:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"NUJkNeRbAS3JQxCgvuxxxTXH1gcy0uVEGLXadb7OphU=","SeedEpoch":1667036}}

Logging Information

2022-03-26T07:18:01.305Z	INFO	sectors	storage-sealing/garbage.go:38	Creating CC sector 356
2022-03-26T07:18:01.305Z	INFO	sectors	storage-sealing/states_sealing.go:48	performing filling up rest of the sector...	{"sector": "356"}
2022-03-26T07:18:01.305Z	WARN	sectors	storage-sealing/states_sealing.go:72	Creating 1 filler pieces for sector 356
2022-03-26T07:18:01.305Z	INFO	sectors	storage-sealing/states_sealing.go:88	Pledge {{1784458 356} 8}, contains []
2022-03-26T11:20:24.593Z	WARN	sectors	storage-sealing/fsm.go:749	sector 356 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: storage call error 101: worker restarted
2022-03-26T11:20:24.594Z	INFO	sectors	storage-sealing/states_failed.go:28	SealPreCommit1Failed(356), waiting 59.405202929s before retrying
2022-03-26T11:41:14.239Z	ERROR	sectors	storage-sealing/fsm.go:34	unhandled sector error (356): normal shutdown of state machine:
2022-03-26T11:44:01.528Z	WARN	auth	auth/handler.go:39	JWT Verification failed (originating from 172.16.0.7:35612): JWT Verification failed: jwt: HMAC verification failed
2022-03-26T11:45:01.628Z	WARN	auth	auth/handler.go:39	JWT Verification failed (originating from 172.16.0.7:35632): JWT Verification failed: jwt: HMAC verification failed
2022-03-26T14:55:10.420Z	WARN	stores	stores/index.go:247	sector {1784458 356} redeclared in b6ad8d84-ab2c-4fd7-b443-f5cd6ca32978
2022-03-26T14:55:10.421Z	WARN	stores	stores/index.go:247	sector {1784458 356} redeclared in b6ad8d84-ab2c-4fd7-b443-f5cd6ca32978
2022-03-26T15:25:11.157Z	WARN	sectors	storage-sealing/fsm.go:749	sector 356 got error event sealing.SectorSealPreCommit2Failed: seal pre commit(2) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
2022-03-26T15:25:11.181Z	INFO	sectors	storage-sealing/states_failed.go:28	SealPreCommit2Failed(356), waiting 59.818708971s before retrying
2022-03-26T15:35:01.902Z	WARN	auth	auth/handler.go:39	JWT Verification failed (originating from 172.16.0.7:35622): JWT Verification failed: jwt: HMAC verification failed
2022-03-26T16:25:16.713Z	INFO	sectors	storage-sealing/states_sealing.go:423	submitting precommit for sector 356 (deposit: 12691768088194580):
2022-03-26T16:25:57.909Z	INFO	sectors	storage-sealing/states_sealing.go:470	Sector precommitted: 356
2022-03-26T16:46:15.339Z	WARN	sectors	storage-sealing/fsm.go:749	sector 356 got error event sealing.SectorChainPreCommitFailed: handler: websocket connection closed
2022-03-27T09:35:46.813Z	INFO	sectors	storage-sealing/states_sealing.go:570	KOMIT 356 c3047d579e825962fd1f796f5e3e45aff3025606d081d097ec7feb2f1194d40f(1664860); 6dd650f628fb4090bb1daa1d6c2d183781b06e736707c0293f224caed19d7736(1667003); [{34359738368 baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq}]; r:bagboea4b5abcai5jtzzubuvqai5a5aqodax4pjptejaihjupe2xbkpemucompqjr; d:baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
2022-03-27T10:41:50.238Z	WARN	stores	stores/index.go:247	sector {1784458 356} redeclared in b6ad8d84-ab2c-4fd7-b443-f5cd6ca32978
2022-03-27T23:23:49.993Z	INFO	sectors	storage-sealing/states_sealing.go:570	KOMIT 356 c3047d579e825962fd1f796f5e3e45aff3025606d081d097ec7feb2f1194d40f(1664860); 6dd650f628fb4090bb1daa1d6c2d183781b06e736707c0293f224caed19d7736(1667003); [{34359738368 baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq}]; r:bagboea4b5abcai5jtzzubuvqai5a5aqodax4pjptejaihjupe2xbkpemucompqjr; d:baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
2022-03-28T07:59:33.874Z	ERROR	sectors	storage-sealing/fsm.go:34	unhandled sector error (356): normal shutdown of state machine:
2022-03-28T08:02:06.211Z	INFO	sectors	storage-sealing/states_sealing.go:570	KOMIT 356 c3047d579e825962fd1f796f5e3e45aff3025606d081d097ec7feb2f1194d40f(1664860); 6dd650f628fb4090bb1daa1d6c2d183781b06e736707c0293f224caed19d7736(1667003); [{34359738368 baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq}]; r:bagboea4b5abcai5jtzzubuvqai5a5aqodax4pjptejaihjupe2xbkpemucompqjr; d:baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
2022-03-28T08:28:07.186Z	WARN	stores	stores/index.go:247	sector {1784458 356} redeclared in b6ad8d84-ab2c-4fd7-b443-f5cd6ca32978
2022-03-28T17:41:19.594Z	ERROR	sectors	storage-sealing/fsm.go:34	unhandled sector error (356): normal shutdown of state machine:
2022-03-28T17:44:00.160Z	INFO	sectors	storage-sealing/states_sealing.go:570	KOMIT 356 c3047d579e825962fd1f796f5e3e45aff3025606d081d097ec7feb2f1194d40f(1664860); 6dd650f628fb4090bb1daa1d6c2d183781b06e736707c0293f224caed19d7736(1667003); [{34359738368 baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq}]; r:bagboea4b5abcai5jtzzubuvqai5a5aqodax4pjptejaihjupe2xbkpemucompqjr; d:baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq


2022-03-26T07:30:01.188Z	INFO	sectors	storage-sealing/garbage.go:38	Creating CC sector 357
2022-03-26T07:30:01.189Z	INFO	sectors	storage-sealing/states_sealing.go:48	performing filling up rest of the sector...	{"sector": "357"}
2022-03-26T07:30:01.189Z	WARN	sectors	storage-sealing/states_sealing.go:72	Creating 1 filler pieces for sector 357
2022-03-26T07:30:01.189Z	INFO	sectors	storage-sealing/states_sealing.go:88	Pledge {{1784458 357} 8}, contains []
2022-03-26T11:19:24.452Z	WARN	sectors	storage-sealing/fsm.go:749	sector 357 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: storage call error 101: worker restarted
2022-03-26T11:19:24.470Z	INFO	sectors	storage-sealing/states_failed.go:28	SealPreCommit1Failed(357), waiting 59.529680603s before retrying
2022-03-26T11:41:14.239Z	ERROR	sectors	storage-sealing/fsm.go:34	unhandled sector error (357): normal shutdown of state machine:
2022-03-26T15:31:06.791Z	WARN	stores	stores/index.go:247	sector {1784458 357} redeclared in 09130f34-eb49-480d-ac99-e39227eeafa7
2022-03-26T15:31:06.791Z	WARN	stores	stores/index.go:247	sector {1784458 357} redeclared in 09130f34-eb49-480d-ac99-e39227eeafa7
2022-03-26T16:40:54.802Z	INFO	sectors	storage-sealing/states_sealing.go:423	submitting precommit for sector 357 (deposit: 12691665787211640):
2022-03-26T16:42:24.656Z	INFO	sectors	storage-sealing/states_sealing.go:470	Sector precommitted: 357
2022-03-26T16:46:15.339Z	WARN	sectors	storage-sealing/fsm.go:749	sector 357 got error event sealing.SectorChainPreCommitFailed: handler: websocket connection closed
2022-03-27T09:35:52.920Z	INFO	sectors	storage-sealing/states_sealing.go:570	KOMIT 357 e2f6eee68dd0a188c5e688adc524c3ac4ac10bacf6a7fe2589db9e065649d365(1664882); 35426435e45b012dc94310a0beec71c535c7d60732d2e54418b5da75becea615(1667036); [{34359738368 baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq}]; r:bagboea4b5abcarwu4iyj53puuzqalzzhiglx7s2welgkuudi3fs5lntawicguriu; d:baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
2022-03-27T23:23:49.994Z	INFO	sectors	storage-sealing/states_sealing.go:570	KOMIT 357 e2f6eee68dd0a188c5e688adc524c3ac4ac10bacf6a7fe2589db9e065649d365(1664882); 35426435e45b012dc94310a0beec71c535c7d60732d2e54418b5da75becea615(1667036); [{34359738368 baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq}]; r:bagboea4b5abcarwu4iyj53puuzqalzzhiglx7s2welgkuudi3fs5lntawicguriu; d:baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
2022-03-28T07:59:33.875Z	ERROR	sectors	storage-sealing/fsm.go:34	unhandled sector error (357): normal shutdown of state machine:
2022-03-28T08:02:06.211Z	INFO	sectors	storage-sealing/states_sealing.go:570	KOMIT 357 e2f6eee68dd0a188c5e688adc524c3ac4ac10bacf6a7fe2589db9e065649d365(1664882); 35426435e45b012dc94310a0beec71c535c7d60732d2e54418b5da75becea615(1667036); [{34359738368 baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq}]; r:bagboea4b5abcarwu4iyj53puuzqalzzhiglx7s2welgkuudi3fs5lntawicguriu; d:baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
2022-03-28T17:41:19.597Z	ERROR	sectors	storage-sealing/fsm.go:34	unhandled sector error (357): normal shutdown of state machine:
2022-03-28T17:44:00.160Z	INFO	sectors	storage-sealing/states_sealing.go:570	KOMIT 357 e2f6eee68dd0a188c5e688adc524c3ac4ac10bacf6a7fe2589db9e065649d365(1664882); 35426435e45b012dc94310a0beec71c535c7d60732d2e54418b5da75becea615(1667036); [{34359738368 baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq}]; r:bagboea4b5abcarwu4iyj53puuzqalzzhiglx7s2welgkuudi3fs5lntawicguriu; d:baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq

Repo Steps

Run '...'
Do '...'
See error '...'
...

Answered by iseriser

Mar 31, 2022

Nope, I have restarted both daemon and miner today. Those sectors still hang in Committing state.
I think I need to try Reiers & Hidde's solution #4086 (comment)

View full answer

iseriser · 2022-03-28T19:03:08Z

iseriser
Mar 28, 2022
Author

I have found that files like unsealed/s-t0178xxxx-357 and unsealed/s-t0178xxxx-356 are stored in my PC1 miner sealing folder.

0 replies

TippyFlitsUK · 2022-03-28T19:25:34Z

TippyFlitsUK
Mar 28, 2022
Maintainer

Hi @iseriser

Thank you for taking the time to provide detailed logging information!

Looking at your logs it would appear that you are seeing storage related errors. You mentioned in your last comment that your are seeing files generated in your unsealed folder. Can you please provide some additional information about your sealing disks?

What types of disk are you using for sealing? HDD, SDD, NVMe .etc.
How much space have you allocated for your sealing and how much space remains on the disk that is currently hosting the unsealed folder you mentioned above.

Many thanks

0 replies

llifezou · 2022-03-29T01:59:04Z

llifezou
Mar 29, 2022

The status is Committing, that is, waiting to execute C1/C2. If the encapsulated sector file is gone, it is stuck in C1. If the encapsulated file exists, it may be stuck in C2.

0 replies

iseriser · 2022-03-29T07:40:34Z

iseriser
Mar 29, 2022
Author

What types of disk are you using for sealing? HDD, SDD, NVMe .etc.

How much space have you allocated for your sealing and how much space remains on the disk that is currently hosting the unsealed folder you mentioned above.

RAID0 4xNVMe drives
[######**** ] 3.165 TiB/14.44 TiB 21%

0 replies

iseriser · 2022-03-29T07:44:05Z

iseriser
Mar 29, 2022
Author

The status is Committing, that is, waiting to execute C1/C2. If the encapsulated sector file is gone, it is stuck in C1. If the encapsulated file exists, it may be stuck in C2.

Those two do not change state for 48 hours. I have restarted miner, workers, there were no other sectors in queue, but they were not being proceeded in any way.

0 replies

llifezou · 2022-03-29T07:56:48Z

llifezou
Mar 29, 2022

sectors storage-sealing/states_sealing.go:570 KOMIT 356 is printed, the processing of the state machine is no problem, the high probability is that C1 cannot be scheduled

0 replies

llifezou · 2022-03-29T07:58:32Z

llifezou
Mar 29, 2022

Does the sector files exist? Or do you observe the log related to declare when you restart

0 replies

iseriser · 2022-03-29T08:02:02Z

iseriser
Mar 29, 2022
Author

$ fgrep 'declare' lotus-miner.log  | fgrep 356
2022-03-26T14:55:10.420Z	WARN	stores	stores/index.go:247	sector {1784458 356} redeclared in b6ad8d84-ab2c-4fd7-b443-f5cd6ca32978
2022-03-26T14:55:10.421Z	WARN	stores	stores/index.go:247	sector {1784458 356} redeclared in b6ad8d84-ab2c-4fd7-b443-f5cd6ca32978
2022-03-27T10:41:50.238Z	WARN	stores	stores/index.go:247	sector {1784458 356} redeclared in b6ad8d84-ab2c-4fd7-b443-f5cd6ca32978
2022-03-28T08:28:07.186Z	WARN	stores	stores/index.go:247	sector {1784458 356} redeclared in b6ad8d84-ab2c-4fd7-b443-f5cd6ca32978
$fgrep 'declare' lotus-miner.log  | fgrep 357
2022-03-26T15:31:06.791Z	WARN	stores	stores/index.go:247	sector {1784458 357} redeclared in 09130f34-eb49-480d-ac99-e39227eeafa7
2022-03-26T15:31:06.791Z	WARN	stores	stores/index.go:247	sector {1784458 357} redeclared in 09130f34-eb49-480d-ac99-e39227eeafa7

But now those worker IDs do not exist.
Files exist like unsealed/s-t0178xxxx-357 and unsealed/s-t0178xxxx-356, but due to worker restart worker ID has changed, but no redeclaring happens.

0 replies

TippyFlitsUK · 2022-03-29T10:15:08Z

TippyFlitsUK
Mar 29, 2022
Maintainer

Hi @iseriser

Thanks for the report. The issue you are experiencing lies elsewhere.

Moving this issue to Lotus Discussions for help and troubleshooting.

Closing Lotus issue ticket.

0 replies

TippyFlitsUK · 2022-03-29T10:22:16Z

TippyFlitsUK
Mar 29, 2022
Maintainer

Updated topic for better visibility:

Unable to Recover Sectors Following Following Sealing Disruption

0 replies

iseriser · 2022-03-30T19:07:46Z

iseriser
Mar 30, 2022
Author

lotus-miner sectors remove --really-do-it <SectorNum>
this does not work either :(

0 replies

TippyFlitsUK · 2022-03-30T19:11:07Z

TippyFlitsUK
Mar 30, 2022
Maintainer

You may find that restarting your miner completes the removal @iseriser .

0 replies

TippyFlitsUK · 2022-03-31T19:44:14Z

TippyFlitsUK
Mar 31, 2022
Maintainer

Hey @iseriser

Did the miner restart clear those sectors that you were trying to remove?

0 replies

iseriser · 2022-03-31T19:46:22Z

iseriser
Mar 31, 2022
Author

Nope, I have restarted both daemon and miner today. Those sectors still hang in Committing state.
I think I need to try Reiers & Hidde's solution #4086 (comment)

0 replies

TippyFlitsUK · 2022-03-31T19:52:28Z

TippyFlitsUK
Mar 31, 2022
Maintainer

Many thanks for the update @iseriser!

That is a great suggestion!

Let me know if the issue persists after trying the workaround.

0 replies

iseriser · 2022-04-03T10:39:56Z

iseriser
Apr 3, 2022
Author

I confirm that solution works!

0 replies

TippyFlitsUK · 2022-04-03T12:02:20Z

TippyFlitsUK
Apr 3, 2022
Maintainer

Great news @iseriser!!

Thank you very much for keeping us updated!!

0 replies

Unable to Recover Sectors Following Following Sealing Disruption #8395

Uh oh!

iseriser Mar 28, 2022

Checklist

Lotus component

Lotus Version

Describe the Bug

Logging Information

Repo Steps

Replies: 17 comments

Uh oh!

Uh oh!

iseriser Mar 28, 2022 Author

Uh oh!

Uh oh!

TippyFlitsUK Mar 28, 2022 Maintainer

Uh oh!

llifezou Mar 29, 2022

Uh oh!

iseriser Mar 29, 2022 Author

Uh oh!

iseriser Mar 29, 2022 Author

Uh oh!

llifezou Mar 29, 2022

Uh oh!

llifezou Mar 29, 2022

Uh oh!

Uh oh!

iseriser Mar 29, 2022 Author

Uh oh!

TippyFlitsUK Mar 29, 2022 Maintainer

Uh oh!

TippyFlitsUK Mar 29, 2022 Maintainer

Uh oh!

iseriser Mar 30, 2022 Author

Uh oh!

TippyFlitsUK Mar 30, 2022 Maintainer

Uh oh!

TippyFlitsUK Mar 31, 2022 Maintainer

Uh oh!

Uh oh!

iseriser Mar 31, 2022 Author

Uh oh!

TippyFlitsUK Mar 31, 2022 Maintainer

Uh oh!

iseriser Apr 3, 2022 Author

Uh oh!

TippyFlitsUK Apr 3, 2022 Maintainer

iseriser
Mar 28, 2022

iseriser
Mar 28, 2022
Author

TippyFlitsUK
Mar 28, 2022
Maintainer

llifezou
Mar 29, 2022

iseriser
Mar 29, 2022
Author

iseriser
Mar 29, 2022
Author

llifezou
Mar 29, 2022

llifezou
Mar 29, 2022

iseriser
Mar 29, 2022
Author

TippyFlitsUK
Mar 29, 2022
Maintainer

TippyFlitsUK
Mar 29, 2022
Maintainer

iseriser
Mar 30, 2022
Author

TippyFlitsUK
Mar 30, 2022
Maintainer

TippyFlitsUK
Mar 31, 2022
Maintainer

iseriser
Mar 31, 2022
Author

TippyFlitsUK
Mar 31, 2022
Maintainer

iseriser
Apr 3, 2022
Author

TippyFlitsUK
Apr 3, 2022
Maintainer