Finalized sector panic, "missing" sc-02-data-layer-1.dat files already deleted by miner

[shotcollin]

I have some sectors that failed recently when finalizing and all of them are reporting storage call error 0: StandaloneSealCommit: Missing store file (or associated split paths). However, none of these files are missing. There's a possibility there was a network interruption or something when they were finalizing but I don't know there was one, and other sectors finalized around that time without problems on my two C2 workers.

I ran into this problem again and was able to get a better assessment of the events that led up to it, so I'm updating this discussion. I was also able to confirm how to (sort of) recover the sectors. Here's what happened:

• The balances in the wallets available to the miner all became too low to finalize sectors. Balances became insufficient between 2022-03-20 08:38 UTC and 2022-03-20 09:53 UTC. This affected sectors in two different ways:
  • The first sector to fail was able to recover and successfully complete after balance was replenished.
  • All subsequent sectors were not able to recover and had to be restarted.
• Sector 1973: completed successfully when balance was replenished. see sector log below
  • SectorFinalized at 2022-03-20 09:46:27
  • SectorCommitSubmitted at 2022-03-20 09:46:27 with message
  • SectorRetrySubmitCommit at 2022-03-20 09:49:29
  • SectorCommitSubmitted at 2022-03-20 09:49:31 with message
  • SectorRetrySubmitCommit at 2022-03-20 09:53:00
  • SectorCommitFailed at 2022-03-20 09:53:00 with SysErrInsufficientFunds
  • SectorRetryComputeProof, SectorRetryCommitWait, SectorRetrySubmitCommit and SectorCommitFailed with SysErrInsufficientFunds all at 2022-03-20 09:54:00
  • The SectorRetryComputeProof, SectorRetryCommitWait, SectorRetrySubmitCommit and SectorCommitFailed errors occurred every minute until the balance was replenished. At that point SectorCommitSubmitted was followed by SectorProving.
• Sector 1974: did not complete successfully. see sector log below
  • SectorFinalized at 2022-03-20 10:39:18.
  • SectorCommitFailed at 2022-03-20 10:39:18 with SysErrInsufficientFunds
  • SectorRetryComputeProof and SectorComputeProofFailed at 2022-03-20 10:40:18 with storage call error 0:StandaloneSealCommit: Missing store file (or associated split paths): /mnt/bn01/cache/s-t01197354-1974/sc-02-datalayer-1.dat; includes stack backtrace
  • SectorRetryComputeProof at 2022-03-20 10:41:18 (no immediate compute proof fail, but all subsequent sectors failed immediately the second time as well)
  • SectorComputeProofFailed at 2022-03-20 10:42:39 with storage call error 0: StandaloneSealCommit: Missing store file(or associated split paths): /mnt/bn01/cache/s-t01197354-1974/sc-02-data-layer-1.dat; includes stack backtrace
  • SectorSealPreCommit1Failed at 2022-03-20 10:43:39 with consecutive compute fails
  • SectorRetrySealPreCommit1 and SectorSealPreCommit1Failed at 2022-03-20 10:43:39 with storage call error 0: failed to acquire sector {1197354 1974} from remote(1): sector not found
  • It seems this last set of errors would continue forever without intervention.

I have two questions based on the above:

1. If both sectors are in the Finalized state, why does only one successfully complete once the miner balance is sufficient? The only difference I can see between the two sectors is that the first one was able to send a commit submit message to the pool before the balance became too low to even do that. But that seems to fail after the second retry, and then it gets SectorCommitFailed with SysErrInsufficientFunds like the second sector. Unlike the second one, the first one continues SectorRetryComputeProof until it's able to complete SectorRetrySubmitCommit. Does this have something to do with the rust/ffi panic detailed in the backtrace?
2. Even more perplexing: once the second sector gets its first SectorCommitFailed error, it goes to SectorRetryComputeProof and fails with SectorComputeProofFailed and a message saying it can't find sc-02-datalayer-1.dat in the cache folder. But this sector was about to be finalized, and sc-02-datalayer-1.dat is a file created during PC1 and deleted by the miner after C2 completes. Is there any situation where a sector in SectorFinalized state would still have sc-02-datalayer-x.dat files stored somewhere? If these files are potentially needed for the compute proof at this stage, why have they already been deleted by the miner? This same thing happened to about 15 other sectors after sector 1974, so it wasn't a one-off thing or a network hiccup or a single missing file. All of the sectors got to SectorFinalized with zero problems, then all failed with the same error about the file not being available.

Once I discovered this problem and got the miner balance back up, etc., the failed sectors continued on in the same loop. The sectors did finally restart and successfully complete with no additional errors after I did the following:

1. Moved the sector files from their location in long-term storage to somewhere not accessible by the miner. The files in long-term storage were the sealed sector files and the cache directories containing p_aux, t_aux, and sc-02-data-tree-r-last-0.dat through sc-02-data-tree-r-last-7.dat. (I don't think this was actually necessary to get them to complete but I wanted to preserve them just in case.)
2. Created a dummy file for each sector in the miner's sealing storage location in the unsealed directory. (e.g. with touch /sealing-storage/location/unsealed/s-t01197354-1974)
3. Forced an update to the miner's index (or whatever you would call it?) of the seal/store storage locations, e.g. by restarting any worker (or the miner), causing the sector {xxxxxxx yyy} redeclared in zzzzzz log messages.
4. Only after dummy file creation and the sector file redeclaration did sectors restart PC1.

old info

From the first iteration of this discussion. I have some sectors that failed recently when finalizing and all of them are reporting `storage call error 0: StandaloneSealCommit: Missing store file (or associated split paths)`. However, none of these files are missing. There's a **possibility** there was a network interruption or something when they were finalizing but I don't know there was one, and other sectors finalized around that time without problems on my two C2 workers.

What is very confusing is now that those sectors went back to PC1, they're all reporting seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1708} from remote(1): sector not found even though there are most definitely files in the locations that are listed for them.

This has actually happened before but I didn't have the energy to debug at the time and I just removed the sectors, but I'd rather not do that and get to the bottom of the problem. Anyone have any advice on where I should look to troubleshoot?

Including what seems like relevant info in next msgs.

[shotcollin]

[shotcollin]

old log from first post

The status log for this sector. I'm wondering what remote(1) in the log (no. 28 and onward) is referring to. Does it have something to do with the lists of URLs above? The first one in each list is the miner.

In any case, the (cache and sealed) files are there, in both locations, and this message has been repeating for much, much longer than any brief network interruption (if there was one at all) would have taken.

user@miner:~ $lotus-miner sectors status --log 1701

SectorID:	1701
Status:		ComputeProofFailed
CIDcommD:	baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
CIDcommR:	bagboea4b5abcaa7m26v4iro4lf5mcvpf7ai7o54zaa4ei46uutuqx2vrojco5ncz
Ticket:		5de96511309db24050f2768d39e9cb5c76ac7057f034ecd7e20914dd08990ac2
TicketH:	1592255
Seed:		50979315c8d9c6397f80f1475a083958140b76e6ccfdb387f7f5f0182a6382cd
SeedH:		1593717
Precommit:	bafy2bzacea53lsovqgtszbk2yisvr7xrsgvsteryqske35bxx3uinrxhv32m4
Commit:		<nil>
Deals:		[0]
Retries:	0
--------
Event Log:
0.	2022-03-01 02:16:11 +0000 UTC:	[event;sealing.SectorStartCC]	{"User":{"ID":1701,"SectorType":8}}
1.	2022-03-01 02:17:33 +0000 UTC:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}]}}
2.	2022-03-01 02:17:33 +0000 UTC:	[event;sealing.SectorTicket]	{"User":{"TicketValue":"XellETCdskBQ8naNOenLXHascFfwNOzX4gkU3QiZCsI=","TicketEpoch":1592255}}
3.	2022-03-01 05:21:55 +0000 UTC:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJYZWxsRVRDZHNrQlE4bmFOT2VuTFhIYXNjRmZ3Tk96WDRna1UzUWlaQ3NJPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoyMTQ3NDgzNjQ3fSwibGFiZWxzIjp7IlN0YWNrZWREcmczMkdpQlYxIjp7Il9oIjpudWxsLCJsYWJlbHMiOlt7ImlkIjoibGF5ZXItMSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMiIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNyIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTAiLCJwYXRoIjoiL21udC9zdG9yZXMvY2FjaGUvcy10MDExOTczNTQtMTcwMSIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTExIiwicGF0aCI6Ii9tbnQvc3RvcmVzL2NhY2hlL3MtdDAxMTk3MzU0LTE3MDEiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
4.	2022-03-01 05:43:14 +0000 UTC:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcaa7m26v4iro4lf5mcvpf7ai7o54zaa4ei46uutuqx2vrojco5ncz"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
5.	2022-03-01 05:43:14 +0000 UTC:	[event;sealing.SectorPreCommitted]	{"User":{"Message":{"/":"bafy2bzacea53lsovqgtszbk2yisvr7xrsgvsteryqske35bxx3uinrxhv32m4"},"PreCommitDeposit":"14932503241492082","PreCommitInfo":{"SealProof":8,"SectorNumber":1701,"SealedCID":{"/":"bagboea4b5abcaa7m26v4iro4lf5mcvpf7ai7o54zaa4ei46uutuqx2vrojco5ncz"},"SealRandEpoch":1592255,"DealIDs":[],"Expiration":2200985,"ReplaceCapacity":false,"ReplaceSectorDeadline":0,"ReplaceSectorPartition":0,"ReplaceSectorNumber":0}}}
6.	2022-03-01 05:46:29 +0000 UTC:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIgoDhoWMp4MukHzuNeLs32Sg0MwQiq4Aw4QTXZg4tJGvMBcaDkAiAfnaQ1rCoUPqrkZFIKNMAvn6qHznbDztQ0d9aeUDRa+wFxoOQCIPhpPTCtdSs7EN6m0Gjx4dalqKq1wBDcSxz66Q97egrrAXGg5AIga/P/qKiPUorkyP9LP6V78933MMUKLQdXpQqt+ZcOCSwBcaDkAiC4KHt9WwYV00ukAsnsV8KSsz18NtOUgjGBJgd/PkfPuwFxoOQCINJFXshrYLuek/NgO6HXE7kt6V6EE0E7SuiFB74Jc0Jn"}}
7.	2022-03-01 07:01:30 +0000 UTC:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"UJeTFcjZxjl/gPFHWgg5WBQLdubM/bOH9/XwGCpjgs0=","SeedEpoch":1593717}}
8.	2022-03-01 10:46:02 +0000 UTC:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 101: worker restarted
9.	2022-03-01 10:47:02 +0000 UTC:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
10.	2022-03-01 10:58:26 +0000 UTC:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 0: task aborted
11.	2022-03-01 10:59:26 +0000 UTC:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
12.	2022-03-01 12:36:17 +0000 UTC:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(2): storage call error 101: worker restarted
13.	2022-03-01 12:37:17 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	consecutive compute fails
14.	2022-03-01 12:38:17 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
15.	2022-03-01 14:59:57 +0000 UTC:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJYZWxsRVRDZHNrQlE4bmFOT2VuTFhIYXNjRmZ3Tk96WDRna1UzUWlaQ3NJPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoyMTQ3NDgzNjQ3fSwibGFiZWxzIjp7IlN0YWNrZWREcmczMkdpQlYxIjp7Il9oIjpudWxsLCJsYWJlbHMiOlt7ImlkIjoibGF5ZXItMSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMiIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNyIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xNzAxIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTAiLCJwYXRoIjoiL21udC9zdG9yZXMvY2FjaGUvcy10MDExOTczNTQtMTcwMSIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTExIiwicGF0aCI6Ii9tbnQvc3RvcmVzL2NhY2hlL3MtdDAxMTk3MzU0LTE3MDEiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
16.	2022-03-01 15:10:20 +0000 UTC:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcaa7m26v4iro4lf5mcvpf7ai7o54zaa4ei46uutuqx2vrojco5ncz"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
17.	2022-03-01 15:11:37 +0000 UTC:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIgrWoI5RJzZiQSz6MDacYgwBu4MIcP1QKI0w8zNVZFETsBcaDkAiAR8fef1eYWYbO+Bc0R/wOe0bd4yQKRZUSpjMKkztiZMAFxoOQCIA3ILZ/dGH6m8drS4UnsZQw0HCsb1ab5fJFk9YRZBCGX"}}
18.	2022-03-01 15:11:44 +0000 UTC:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"UJeTFcjZxjl/gPFHWgg5WBQLdubM/bOH9/XwGCpjgs0=","SeedEpoch":1593717}}
19.	2022-03-01 15:31:01 +0000 UTC:	[event;sealing.SectorProofReady]	{"User":{"Proof":"lCprl4xTb0Tx2QTVbPJlD3dYu4wAz6KQoyCxU3iOMf3/QUWqTZKUXVQzpzlDVelkoBwjXA9HqBWL5TAn2W9jEzw7s26iiwh+UZDNnVOIizqNaA79KR9Iacj6w54xlVk7EKdbukauygYForgf6Bk3GDGw2Onrq745sTzlMu0lMSaGnJjaEGgIbjVuLW+D/Brnrfm9+LK0rLaeLlA8JLftQmL1JoInKMYAoq9wCdNQWGIv2xhdQ8+ftGLV1S1h+2OHpjeD1AKyAolLAFF8HlNeyIcBBl2sUt+0LCnsCiiKJEG1gNFMfMEFmAYnqx75l6C1qdfin2xoayrHvdPUL2gM7Zi/w2ieoJ1q3+Fm4humAFu2Y3EBlVDuRhO0I1tsOZq6GJi79X6Ck3l4C0ve23b3bgqQjHZ5669pIKBGq7s/3Ojpr2pyVozsXMKl67HxRVbkiz4WBJFC0BKrerL9BDX/aZX5bVzzzAY4lnhhbt4TamMsrKJoxDPCN2syh+KOaLZ/j3yeCFq42JBtlzjaKRYjTO+g0IaRugtfr8Y30r7qTC/yQbpKxsl07GSbGUW8M1AzkDZNPLy93U9XwXfz0+3NeSM38xSn4RiiFZWxoVafEateSWWSkpGB7uKacRb+srylBH1pasVtwknFa1gTAS0/paiHpXaL2gXaUBUb/1bkntA4W7hhj5O+JJCj7Wg6GxVti91wr+1IsaqQvjE58WbFnS8NS2ZlrkzSjujbhCgvJ7UzcBbZ8C/ZvXxrv5UdGJZvgi3bm/qO+RJwKlyKIFc/cDA3RC1Rm1/5Vs2BpYWUvZaozjaZUoi9yL7zOEBpJ9//j7aB6QD2ixmMKkVYPSSAcv8rOvZEvQAhM0GqOe39NKydQGFrunQv4mdxvVGRVbkvDxrImJQFn5ZNi/mZxHwgjkUXo8iGlbY8JxvM7HJYeKFQFfVZRK2M6BUWfb+Ckrk3hl9Le6DtqfOZlFajkI7cNj40f7Mqo+qQZA02YWuU1az5DQSrtLaWtY8o5Gb5xbQyrrQ/MYPXSooW3xGy3cf09NB7cSl5eh+yazwDco5mDMh2O8UQuPWTCnwRMHZw90ujk8r17xjMvlrajRCO7I2WgZ6mfciKGFIpR8YN15jkF6h/7PpZsaglYoNFd41it60IBEoB85VLeZjuQguns7uxOlvTpsHGN91GRKpFpxgSlYmY1wZe+Sfw1i/W6q7U04mGh5Yt5fzDmHpZHXeFEGR4gMPuYxhzWOoO5gXXC9s/4EBxOVEWwYGD4gss4sSnkcrvpAF41KeWp32ZDIYt9K2Bpv4uxbMyPFRf2j17PdEgQCEFzdOw2NRrhLeIdDulJ+5oi8TJfz0J0es/z7IJtI9y9wXYGdMBkTA9DLmuiVFHO4TQEKrjZDpBPgemASa8rYsHCU0Aiq8+zTOAipyQq6SPYnl6KJXC5ycZkUtAJqI6KQQ9LGGi8asqGaDP/3ka6DOzhBl3s2AvWEHlqqzh1nvpVYuU64Yn6+XZq+Fd6wLo0P3lIICUBEKeRltn0q5NqMououoVc0h+RjNsGGxj3ELI4gshsVOTZS4mUdaUH+MeUj/OMIa/JHsicbttUOL+2e+Crmgg5LbvtTIUAbny1fGcRnRkOKHy7YTSdnWDfOs5kr4eiuJ8htA7MtFusK0LoSV1A+rUFywiJ4QJSwb6ZAl5bRWc2WXqEFjdJkfa4JGKyZykalGgTUCSsjgHwhqUYfFgqMSgdooz38icdE5hVNfVSSxmiTF19v8KQ8vpl8MYTU1yNOJOVLmg92mjQLDKuhFtkWxgbYGV21SSCiEAKN7rSssyCMvUf3iRSBOTCw6KfIPJEnXRQahoDH79FkG9sbczsNgR5LHNv6oj7KsLxVoLi39WS3jrsOdE2E87ROyyY2Er6Fs9JAXJlY7bXy4vkErCBgpN2Zg8y0HNMcQaMtn5uT7OvkPvyczrmydQnjvk1Ua5ioCDlnZKqL2bzl1FQjzbpXTxoIsotF+jEBVz/urr/lipbgw45UfxvmzYMtBf+L8miFmShzMYniqBtf3TFj02kQTuXfX0R804NVXWaOHpJgJPYUoiqzfo2woUCR48TeCVOsoH+i0bSx9vGDK+4v+UmR+Q1xZ5XBqMk+gFleXBpsjWpdR9GQ/fsNoPbTJUUkqFuiACTr7LWQukoe51rGnSExA0s4w8jZQ1lbFUUbLN3C46+2MUK0zdLxJGixTzkSKSrnLHL6onlMUefinpK3u9kDVVtmjU8ygV9dwDogC/gWC5uVIGp6/kT3uY4sBDjOeK5XKJyFSkYKjsXCxNl0wTuMYDhisXgcJ7UcRfuSs79JfAzvBB8tdws90udF8/vihQREnYIejz9wshfDQYL6Srh5LNcmKKaFTwDARV2iJvxOaEF2QVBApbG6dAKUhPYYhCZxNoX8Gfg+vbLkb6uZf5CcRwnsk++02IZ7QXdIOofj4mv2VnwUk83lj2WRPl56JNu2uZZk9Q6jYl2BzAMwevt844t7KDWbgj0jKoTQqnzPow2fZzW/VX5bKpN0oWOGq70lfxzqyVusREdcJp7d5F"}}
20.	2022-03-01 15:32:16 +0000 UTC:	[event;sealing.SectorFinalized]	{"User":{}}
21.	2022-03-01 15:39:06 +0000 UTC:	[event;sealing.SectorCommitFailed]	{"User":{}}
	pushing message to mpool: handler: websocket connection closed
22.	2022-03-02 00:48:27 +0000 UTC:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
23.	2022-03-02 00:48:29 +0000 UTC:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(1): storage call error 0: StandaloneSealCommit: Missing store file (or associated split paths): /mnt/bn01/cache/s-t01197354-1701/sc-02-data-layer-1.dat

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
   1: filecoin_proofs::api::verify_store
   2: storage_proofs_porep::stacked::vanilla::params::Labels<Tree>::verify_stores
   3: filecoin_proofs::api::validate_cache_for_commit
   4: filecoin_proofs_api::seal::seal_commit_phase1_inner
   5: filecoin_proofs_api::seal::seal_commit_phase1
   6: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
   7: ffi_toolkit::catch_panic_response
   8: fil_seal_commit_phase1
   9: _cgo_24921690cad9_Cfunc_fil_seal_commit_phase1
             at ./tmp/go-build/cgo-gcc-prolog:1499:11
  10: runtime.asmcgocall
             at ./usr/local/go/src/runtime/asm_amd64.s:667
24.	2022-03-02 00:49:29 +0000 UTC:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
25.	2022-03-02 00:49:29 +0000 UTC:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(1): storage call error 0: StandaloneSealCommit: Missing store file (or associated split paths): /mnt/bn01/cache/s-t01197354-1701/sc-02-data-layer-1.dat

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
   1: filecoin_proofs::api::verify_store
   2: storage_proofs_porep::stacked::vanilla::params::Labels<Tree>::verify_stores
   3: filecoin_proofs::api::validate_cache_for_commit
   4: filecoin_proofs_api::seal::seal_commit_phase1_inner
   5: filecoin_proofs_api::seal::seal_commit_phase1
   6: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
   7: ffi_toolkit::catch_panic_response
   8: fil_seal_commit_phase1
   9: _cgo_24921690cad9_Cfunc_fil_seal_commit_phase1
             at ./tmp/go-build/cgo-gcc-prolog:1499:11
  10: runtime.asmcgocall
             at ./usr/local/go/src/runtime/asm_amd64.s:667
26.	2022-03-02 00:50:29 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	consecutive compute fails
27.	2022-03-02 00:51:29 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
28.	2022-03-02 00:51:29 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1701} from remote(1): sector not found
29.	2022-03-02 00:52:29 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
30.	2022-03-02 00:52:29 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1701} from remote(1): sector not found
31.	2022-03-02 00:53:29 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
32.	2022-03-02 00:53:29 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1701} from remote(1): sector not found
33.	2022-03-02 00:56:11 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
34.	2022-03-02 00:56:14 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1701} from remote(1): sector not found
... [removed repeating lines] ...
293.	2022-03-02 09:08:34 +0000 UTC:	[event;sealing.SectorForceState]	{"User":{"State":"ComputeProofFailed"}}
294.	2022-03-02 09:08:34 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}

[shotcollin]

[2022-03-20 23:14:12]user@miner:~ $ lotus-miner sectors status --log 1973 | head -n 40 && echo -e "\n[...]\n" && lotus-miner sectors status --log 1973 | tail
SectorID:	1973
Status:		Proving
CIDcommD:	baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
CIDcommR:	bagboea4b5abcbml54rl7xcma62r54q5bh3tzkrdmsmxccfdxmkbcn4g33ab7m2sd
Ticket:		4d4a736cea40dc1052053824212ab8102a0e757f3fb396c079f6b05a0c881dbd
TicketH:	1647252
Seed:		dd25b7009f9feb01c007eff38f420c0178a75a49b1af85996069b59b1fed4a23
SeedH:		1648693
Precommit:	bafy2bzaceb5hfakbgqxhpedqrgrnsbrnbgor6ofy3gpfhbv3gdktviylgofdm
Commit:		bafy2bzaceau37khq3zsa3mdbzv4c5hril3kci6e6kxypuftyzjglrclcnjv26
Deals:		[0]
Retries:	675
--------
Event Log:
0.	2022-03-20 04:35:01 +0000 UTC:	[event;sealing.SectorStartCC]	{"User":{"ID":1973,"SectorType":8}}
1.	2022-03-20 04:36:14 +0000 UTC:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}]}}
2.	2022-03-20 04:36:14 +0000 UTC:	[event;sealing.SectorTicket]	{"User":{"TicketValue":"TUpzbOpA3BBSBTgkISq4ECoOdX8/s5bAefawWgyIHb0=","TicketEpoch":1647252}}
3.	2022-03-20 07:32:33 +0000 UTC:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJUVXB6Yk9wQTNCQlNCVGdrSVNxNEVDb09kWDgvczViQWVmYXdXZ3lJSGIwPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTczIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoyMTQ3NDgzNjQ3fSwibGFiZWxzIjp7IlN0YWNrZWREcmczMkdpQlYxIjp7Il9oIjpudWxsLCJsYWJlbHMiOlt7ImlkIjoibGF5ZXItMSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTczIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMiIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTczIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTczIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTczIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTczIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTczIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNyIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTczIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTczIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTczIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTAiLCJwYXRoIjoiL21udC9zdG9yZXMvY2FjaGUvcy10MDExOTczNTQtMTk3MyIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTExIiwicGF0aCI6Ii9tbnQvc3RvcmVzL2NhY2hlL3MtdDAxMTk3MzU0LTE5NzMiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
4.	2022-03-20 07:51:19 +0000 UTC:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcbml54rl7xcma62r54q5bh3tzkrdmsmxccfdxmkbcn4g33ab7m2sd"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
5.	2022-03-20 07:51:20 +0000 UTC:	[event;sealing.SectorPreCommitted]	{"User":{"Message":{"/":"bafy2bzaceb5hfakbgqxhpedqrgrnsbrnbgor6ofy3gpfhbv3gdktviylgofdm"},"PreCommitDeposit":"12638302004531942","PreCommitInfo":{"SealProof":8,"SectorNumber":1973,"SealedCID":{"/":"bagboea4b5abcbml54rl7xcma62r54q5bh3tzkrdmsmxccfdxmkbcn4g33ab7m2sd"},"SealRandEpoch":1647252,"DealIDs":[],"Expiration":2255982,"ReplaceCapacity":false,"ReplaceSectorDeadline":0,"ReplaceSectorPartition":0,"ReplaceSectorNumber":0}}}
6.	2022-03-20 07:55:00 +0000 UTC:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIgVN1WBuXNDLH8t2AGWmDAOE3MMbvwy/4Ddt4Ut2R13QABcaDkAiCMXS5x1NmiJRc2iJzkIshHbjCEeiIALT0My+AeUoWFYgFxoOQCIP/E1ZfdTjeNGFRhKvtyGTErQkrgK2ZVv+RLVRBgIovqAXGg5AIgbYvuD1hSLsAt1DE1zFOgU+mrr3IcBX2oYpjF6LZePngBcaDkAiAVn+zYWjKEx8FBuuVVNUDfhYHDgaqoEc7wXoP5IeIX9g=="}}
7.	2022-03-20 09:09:30 +0000 UTC:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"3SW3AJ+f6wHAB+/zj0IMAXinWkmxr4WZYGm1mx/tSiM=","SeedEpoch":1648693}}
8.	2022-03-20 09:44:53 +0000 UTC:	[event;sealing.SectorProofReady]	{"User":{"Proof":"h7VP7bakKXuImTLZ1keHdF7s64AZwCUfItwEjfawJXSjQbZwydGt/ilSLHezI/HTlnMNcmHSbEcbl5vrTNDHC3Dc0+zTj9MzKzKQ+vsKbHh1FN62s7PgkvrVt1VanIwOEP39+YpuvutUphj0SOR5XXGdX/QFqMtZJjiovHa4b86m7sFfFYgC9D+LMbvM59M1kZnrOV8+czLb6lq0cPMDkzziVsJY/DKOPvNb/DlqsMYlolvItvDBSIqKJJ/dcOdqoVPLpput9KnDup4+v1IpXImLU51Vu6ipMEQWq5lErYzTb9e6nAq29Gd1LPDHS6T4pF9zXX1j/OTSuXraqY54NWOWwHzhU5THxny4fF5mpzCTSzjKFu02iSPcpCmcTktEGX7R3gjTPXsbxHtql66mIBG2yCvN3CVNjEyk1ZBrS9wxf4yxj5UEkf8CmDjEUls7r2MFU0L32wa9djj4fJhrPKz/cdikEZbAuWUz91BnmgR07lC7WrDTRTd7vRISDVrQsGx7GqRrehYt8PDVglJwb0sY1G4XZfdIQEDnFChcMRP3kl5u/EXptZW3WrYO4J0goz4PrRQJHypNQKh6axYh9EBvWRuzR0S1E5/NpuCIVep/+TbYv0dhND/ZogM2sBfeAAmSkzBdmmCqLZRYBv3bGIwQ55Ad7AdPeNThDrbT2c+oG9kpFjdIeFNuysBaP/1BoSoWh59JZXETq+oQRznHdlaVsFtThBeld7xlU81xTAxeIXVWpcdel3Re+0BcDPuhi5cX73ihhoJYoyTlh5YK8V+xnmEm6sgUCC+3FGyVz8vI/hpJBUiu0L14RxMzjT/vmUk+fMJVYsfr+N763QQ3Nou9fB3K6AyTJXH1tiEiBU8v4X3ABopS2Tlw4WIWkTk6ADziGRSDsayDR6lmIPLjsFJMz0xg5AI9ePnNKOFRy8whDKrKoos8H27bY87+AZv1gbgnfCPcI58ntq3uyVEa3795QRIOqnC/V17sLkWc957X4eBeRSZSCuRT73/jAuPisTS7QZI6QKp/DyPYHdq44KdsxmvVlUDmB6qRSQ9QwNUqEEH2MlPDDZYStDmsuAusol1mVqDPHnD8tB/PFYoI73Yy4MY/iH4TdsDPcDspsVJuXEQs0R9059GtxJ+xCAbUDu7+pxeS6SfxTtsyoxoF1yhdnPsURqGZDO5Dw6kAPcshyOUYL/jBo5pyghOWNcUXoZ5yzqyKXIsNcS9JaCWU4J+THYYok51m/Ezw7/h9a6BQG30M6+nCuEC08YHeR4Cwl4WeVQKkHe09myf80WMKb051EShICmhSBYnhrFIw53QZneqKvPtN5Nbvu3Y2boYfrK68xyGiroQ0eeHnI8Owh3s8qw93vyBAvYDUxCHsEcyfnA1JAcNfY1KSVHAkjlAbB+g3UHKvVepsFrQFtVvHd4VumysUAVAGIcfOWTYtiEEdXiIFtl8jM2jfh9ccXNhJpIK8/7Ou3v9Yd/gEUUYWtueH3MNi2f1l8mtXNhSPbXTjXEWaPjBSiPiUbESPWEa9pI3CACqEH5L/4ETsOlxei0u4L6cv5QaYrA2HlYxrDsY+q83LYtNPH/03na2+qRE8mbuG8EHaSJc8mMNasorLx1+tovS9YHcrU8rA+MixVYmpbquxkmfknDaY4xeYfK7EBeSJtzYxkXNHi2YBU15mUBbgN3bwBLRXdxw77cQynLh7Ep5FgrngUp0H4dHx/BnRt/d9UUHirKXthwVlTSlj5zLdB2ZlveMzcy4Bzf99gwT3VkHbHa1AKC7smFR24QYDpv5rWmZEczYoVK+L4oJ2pcU00WMMyWv8bpVQ0PL1jmdCDStPxZulGe00aNOD8QCFpxR/o1h8lo19uQUQqyv1P4CoT8TsWgNevjblZhQYFjaJj3p54eDt9SHHL7K8/g7gEwXxaE/QkySzUByLvvKRLC561rJRcp7/OdlkdbgydQmEgj64H3PFDy/Uabao8Ns4qfl99MuJvwalWQzFlxaMAf5Zx8gGNG6A7KM6ZdqWU9b2b+cK1C4Li+/GdXfFKRKdk2zD9waqcX9Vg04ib3s/l/LEHYCdQoOqkwzIpzYyaR6HuFS+gXus75Oydjta4PPlkqxCfdry9A1UXv2TtqyyyUM5/efYc1n/mDOMyLkTSpr4trQgATGmEARgm5o2kvLrGL+QfqwyuABXAE76976o69DmV4JTRcNe+b1vr9JxCXNKlkaE3XtE4LB0zfx9A1Jrh2DppD2ojCCoBkeUwlGPfkETRAQXXItKqs9cv+eO/w+S1JyO6QA2w9JuqqB8yFuGk/iZR5uEtaaZ+1ZbiSRX7B0whGYloSam0CJ6Ycm73mGdWPIn9rjKaH45DkAhFpSAhufU6qtQXYLsmvSxpTC06m4PThToTJ5HZtspD7TkmVTTlQHRWl7QwsSH38qoCp21EqJYEhsEcX0NlKhI4X6YAHGnRwDmIO1Sr2dW3lP09ihyFhEFgHP6+IHpH4v74jfOkkiCQf3qIKdw0n10lFJrnuVGoBciRYknk/BAvN5Klyp8vFdv8hf30aqa4XckJsOj"}}
9.	2022-03-20 09:46:27 +0000 UTC:	[event;sealing.SectorFinalized]	{"User":{}}
10.	2022-03-20 09:46:27 +0000 UTC:	[event;sealing.SectorCommitSubmitted]	{"User":{"Message":{"/":"bafy2bzacedeb4ya2tv2xgd7ba5qdy6qdafuatmgg2jm67xyqg4gbbcjvrar56"}}}
11.	2022-03-20 09:49:29 +0000 UTC:	[event;sealing.SectorRetrySubmitCommit]	{"User":{}}
12.	2022-03-20 09:49:31 +0000 UTC:	[event;sealing.SectorCommitSubmitted]	{"User":{"Message":{"/":"bafy2bzacea2qt5zhozohkgdduk2spk4rdjqwbs3ghkarwrkfxt2lxntrwghzm"}}}
13.	2022-03-20 09:53:00 +0000 UTC:	[event;sealing.SectorRetrySubmitCommit]	{"User":{}}
14.	2022-03-20 09:53:00 +0000 UTC:	[event;sealing.SectorCommitFailed]	{"User":{}}
	pushing message to mpool: GasEstimateMessageGas error: estimating gas used: message execution failed: exit SysErrInsufficientFunds(6), reason: failed to transfer funds (RetCode=6): transfer failed, insufficient balance in sender actor: 142261778438441955 (RetCode=6)
15.	2022-03-20 09:54:00 +0000 UTC:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
16.	2022-03-20 09:54:00 +0000 UTC:	[event;sealing.SectorRetryCommitWait]	{"User":{}}
17.	2022-03-20 09:54:00 +0000 UTC:	[event;sealing.SectorRetrySubmitCommit]	{"User":{}}
18.	2022-03-20 09:54:00 +0000 UTC:	[event;sealing.SectorCommitFailed]	{"User":{}}
	pushing message to mpool: GasEstimateMessageGas error: estimating gas used: message execution failed: exit SysErrInsufficientFunds(6), reason: failed to transfer funds (RetCode=6): transfer failed, insufficient balance in sender actor: 142261778438441955 (RetCode=6)
19.	2022-03-20 09:55:00 +0000 UTC:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
20.	2022-03-20 09:55:00 +0000 UTC:	[event;sealing.SectorRetryCommitWait]	{"User":{}}
21.	2022-03-20 09:55:00 +0000 UTC:	[event;sealing.SectorRetrySubmitCommit]	{"User":{}}
22.	2022-03-20 09:55:00 +0000 UTC:	[event;sealing.SectorCommitFailed]	{"User":{}}
	pushing message to mpool: GasEstimateMessageGas error: estimating gas used: message execution failed: exit SysErrInsufficientFunds(6), reason: failed to transfer funds (RetCode=6): transfer failed, insufficient balance in sender actor: 142261778438441955 (RetCode=6)

[...]

2708.	2022-03-20 21:13:49 +0000 UTC:	[event;sealing.SectorRetryCommitWait]	{"User":{}}
2709.	2022-03-20 21:13:49 +0000 UTC:	[event;sealing.SectorRetrySubmitCommit]	{"User":{}}
2710.	2022-03-20 21:13:49 +0000 UTC:	[event;sealing.SectorCommitFailed]	{"User":{}}
	pushing message to mpool: GasEstimateMessageGas error: estimating gas used: message execution failed: exit SysErrInsufficientFunds(6), reason: failed to transfer funds (RetCode=6): transfer failed, insufficient balance in sender actor: 26558335036206524 (RetCode=6)
2711.	2022-03-20 21:14:49 +0000 UTC:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
2712.	2022-03-20 21:14:49 +0000 UTC:	[event;sealing.SectorRetryCommitWait]	{"User":{}}
2713.	2022-03-20 21:14:49 +0000 UTC:	[event;sealing.SectorRetrySubmitCommit]	{"User":{}}
2714.	2022-03-20 21:14:49 +0000 UTC:	[event;sealing.SectorCommitSubmitted]	{"User":{"Message":{"/":"bafy2bzaceau37khq3zsa3mdbzv4c5hril3kci6e6kxypuftyzjglrclcnjv26"}}}
2715.	2022-03-20 21:18:00 +0000 UTC:	[event;sealing.SectorProving]	{"User":{}}
2716.	2022-03-20 21:19:13 +0000 UTC:	[event;sealing.SectorFinalized]	{"User":{}}

[shotcollin]

[2022-03-20 23:34:11]user@miner:~ $ lotus-miner sectors status --log 1974 | head -n 80 && echo -e "\n[...]\n" && lotus-miner sectors status --log 1974 | tail
SectorID:	1974
Status:		SealPreCommit1Failed
CIDcommD:	baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
CIDcommR:	bagboea4b5abcatmpscsrpfuatq6lrilrr35ctxmjdsds75bfntjxq2evfuvnfjrs
Ticket:		b69bd1d4ba7850bd8e7c33d7f90eda43a034016b2d090dc31da7f934cfd21a43
TicketH:	1647372
Seed:		3c7d3e935b77f7a9c2dca35ef4274c81239ad25b2deece3cd64481df541810a8
SeedH:		1648809
Precommit:	bafy2bzacecmhiqwg6rhdonfgz7p7hscjbn4mchpi4emq6frbs726kegnyrc2a
Commit:		<nil>
Deals:		[0]
Retries:	0
--------
Event Log:
0.	2022-03-20 05:35:01 +0000 UTC:	[event;sealing.SectorStartCC]	{"User":{"ID":1974,"SectorType":8}}
1.	2022-03-20 05:36:12 +0000 UTC:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}]}}
2.	2022-03-20 05:36:12 +0000 UTC:	[event;sealing.SectorTicket]	{"User":{"TicketValue":"tpvR1Lp4UL2OfDPX+Q7aQ6A0AWstCQ3DHaf5NM/SGkM=","TicketEpoch":1647372}}
3.	2022-03-20 08:33:42 +0000 UTC:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJ0cHZSMUxwNFVMMk9mRFBYK1E3YVE2QTBBV3N0Q1EzREhhZjVOTS9TR2tNPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTc0Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoyMTQ3NDgzNjQ3fSwibGFiZWxzIjp7IlN0YWNrZWREcmczMkdpQlYxIjp7Il9oIjpudWxsLCJsYWJlbHMiOlt7ImlkIjoibGF5ZXItMSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTc0Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMiIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTc0Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTc0Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTc0Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTc0Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTc0Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNyIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTc0Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOCIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTc0Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvbW50L3N0b3Jlcy9jYWNoZS9zLXQwMTE5NzM1NC0xOTc0Iiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTAiLCJwYXRoIjoiL21udC9zdG9yZXMvY2FjaGUvcy10MDExOTczNTQtMTk3NCIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTExIiwicGF0aCI6Ii9tbnQvc3RvcmVzL2NhY2hlL3MtdDAxMTk3MzU0LTE5NzQiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
4.	2022-03-20 08:49:20 +0000 UTC:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcatmpscsrpfuatq6lrilrr35ctxmjdsds75bfntjxq2evfuvnfjrs"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
5.	2022-03-20 08:49:21 +0000 UTC:	[event;sealing.SectorPreCommitted]	{"User":{"Message":{"/":"bafy2bzacecmhiqwg6rhdonfgz7p7hscjbn4mchpi4emq6frbs726kegnyrc2a"},"PreCommitDeposit":"12631067636958772","PreCommitInfo":{"SealProof":8,"SectorNumber":1974,"SealedCID":{"/":"bagboea4b5abcatmpscsrpfuatq6lrilrr35ctxmjdsds75bfntjxq2evfuvnfjrs"},"SealRandEpoch":1647372,"DealIDs":[],"Expiration":2256102,"ReplaceCapacity":false,"ReplaceSectorDeadline":0,"ReplaceSectorPartition":0,"ReplaceSectorNumber":0}}}
6.	2022-03-20 08:52:30 +0000 UTC:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIg3JeEkPWMooEBXnkUBdtuPNLgh/yDYXfns02jvIMeBZcBcaDkAiCxMPwE0aPCzeJKmKDl/u8ADNKdF0VZEcGXKj6Jw14MGQFxoOQCIFo6OtvsA20ZPTFXukWRFApjKLX/ph74KXVHABG+/Y9qAXGg5AIgH8DeOkJNoNmSupKMfqydQPnhCU8uiVLWlvk2XqYm94kBcaDkAiDlLeucmt72+VBs0Oo39gu6dYaJjSjx1Oyy0hXHULl0PQ=="}}
7.	2022-03-20 10:07:30 +0000 UTC:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"PH0+k1t396nC3KNe9CdMgSOa0lst7s481kSB31QYEKg=","SeedEpoch":1648809}}
8.	2022-03-20 10:30:08 +0000 UTC:	[event;sealing.SectorProofReady]	{"User":{"Proof":"odvMqJ6/kuFY0vFVrJt8aUQXUFYiu32U2ACfJ9GAyKLLI3W1wiC97KUQWQoahDodsOSLpDERxR494FrYsozt0YTzbYfkwlcDUKjqruasUEgEaTYMDK0xjWMJG4XDW6mHFjGUL6vVNxYewaidOFx8Be7mhAhKSzlqrTRoaslMpMaWetQdcyRNLTwf3Kc64xkBgB9BzIhJVPlL88qaiKxnNXLYygEtbwAvA/B60vwwSTjWkGagh3e+7WKM9+IAAjsskowztF7atTBG+BfdBo5jMcFIJWmcqbdLQNOhtcfMIdxh6Ko6T+XDClMF59jemD1bqRy5/exOmvvk6P4dIEhHapfWl4fHlkAdoS3qvumZN9tEE+CIzWXfxV1YJSEZ/L4qBn/QpfgY3MtpJzjJpAO/igdgwjfAbkmp5vdTaPuOzYlKRF9w/7IN4oFBqZaPHTPFhNCJ73Kj/o4IC4mSUSF+5IRllxidEUXDAsPC5kR77o1sRxSzJcQb6yovm+ShaVvBhmGeZQbcd4jCBkzoYqST8uvfUyXdq51yVus6lelwI7CBYYm/r9A+d2SbUf4XAJhCp5p+ZlFz/hQ35WR/Myy+89JYupltBbUrWDB/ptrRdktDmiRMiYs64/Tw3HawXkFTE77AwTpjVsNEZ+LjEuL7zZwA2BBUNDRxJYtP4AEaAw5+DAHua0SGm2dX3XJ+pi76pdk2pdtTqWtUIf97Pnk5UR4OO4w6bswIWwfyINk/bxRIaM3rpSmanrIJzB7KVul7gyn2QbtRPqXZ6+5BdDUTSod60ba3Rgt9oSkZZEJBJAeIm/fK9H8eBiRj7Ry5Z0NCqjGnKSFAjTZmyqay/t0dzH/8bhNC2hLVX9ke6K2XahDOEe1+j/EB/ukHbqN6x4woGUrPTQBrNdTcxrbW00W5g7bjvcaYGxJctrjPGJ7L1OPFyaY+oYRXCKDBTNc1Jqqkg1UkU6JdbEv7wHSoGSoO32J1F8jQE4qIDSMnKQjCA/rx/kKfWiE0NDn+73N2VXcJrnWTGf4SwTKge2sCf9vsF8XPuwbIH9lClWETaP+Ie9O8SpHYkRqulYR3n6N0VHWxpcUvdHr0zQUU5kTU5JM9aLubyUYTB/TAHG8TwOtaR+iaCt9uECSsQ3Krli6mncJXC8VjOVjtpPpIWwV3+3K97AKL8vnsdYrY36lVjZT3SfwFUMzMYDn5EhIhuXBaJnwomYFogGV0SlzCys/yYGwwraduY4AxAv5hUvpk+Of4Z34Eb4DEW1OorCnFAy2x0ahNkupo0UU/f+TWZ5/hpCXhGWNeSYRrO+upeLd3/ZAiuUiy/jKexUgag3nXCX5/4ZuXibB5HLNM6H2xaYbf/espx/gxSvhK1olfGyRf3tuBSOV+SaI6zUYwgMQvDx0+U3xuFWAYbTrWyaWKWb0t0uLzA454LxlGJW6xllza33+QCY+gZ8F+BOPOmmMvdODG/srsqSFxm7qNavW4TF8n4tJxN89bTjn9GuXcB27/ZLmsOQy6pcB7F4CwGTKPKCe2HzmuktDRadk7m00qTBAtUrZw/3Yu7PEXN7IHcbYQO6Q2RftyOq5RbgZYUH74dIBnixI7qGEffEKvwh3CCdEGRjuxiKpDg/+R5xu9xNVvhPGQQ4shArbhNKPCS1zdXKFrY+DLCDmnsiyPMlvqA5RJVETEevT6AFZVNehyzQ+3Z+jEmGmRQB5ecL/eCfrY0Aq+nRYOoASS9tQ6AGUSRNYtDGy5AVKJfxalydBUh0GP9fVc0J5cbNL76ZxtnsEszZLxiI6WjBpEGgOVFy3gJXpqAMPOLWHOIbRVqrB8tYNE6zYX6QH3vLR5+V9igeVSYQbdGjN8sCcohSMKeNysKTlqBW8kxaAajpSshBeEOrH7Vp9dGi/LMHFmuSnED2TPN/OwBdDRCQGUP3PGJVmYc0up98zlfXR6InALpQTyvHdhmYUwqJCVzi+u90PfsjD8GwHdYxBJoHKlzqmtJ1YFm2UbU6kK7/w6FK0QFcNQWV6PYqdDKlpB5AmCV4mYDVD8vz4e0i7eiJ1VC3izc5uQjcmmygxzINvZHAErFDxmZdzMI1HVVycMaj4dmnsNkzy3AHmdbSOvg4v3i0GEkOtZ8WrHL75GDSGlij3V0ZbPqpNFQiryvFAtDAlTAsSsXabDcQ8zFSHCDzJ3qGltfEFP2lLohFInNnCNz+IZhJxmdBB7PUcTxUrM/HBEboQuX7KLnkGFP8ewqKc680bgx13thkKco0RFqfM0fpLtEkxe1hHIyTUgdvo+STveMUFFO9vZqg+PMd6WmG3C62c4+Yx2pk5B2z8RS3VB7viRFpcbPg5+sbszKNKF/B7pSFpKfAqULFwio1STsL4IkcYoNdffXa2AiH+s3JPEywFOX5DmQP2dsFx11nq9MDlHjq/yZdmbS35Wv0kJF0veHvl+ZnyCz0zQoe6DQDA32aTb0ph72ESf2jlzy4B3g6A0QGHoUyOd34OlnrIbkKECg/CUrJ/ub+8sk5+w5vijjrmNIKRIbT1b4EO80FDrFdQcHtApxJkJMp7rzrUf"}}
9.	2022-03-20 10:39:18 +0000 UTC:	[event;sealing.SectorFinalized]	{"User":{}}
10.	2022-03-20 10:39:18 +0000 UTC:	[event;sealing.SectorCommitFailed]	{"User":{}}
	pushing message to mpool: GasEstimateMessageGas error: estimating gas used: message execution failed: exit SysErrInsufficientFunds(6), reason: failed to transfer funds (RetCode=6): transfer failed, insufficient balance in sender actor: 142261778438441955 (RetCode=6)
11.	2022-03-20 10:40:18 +0000 UTC:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
12.	2022-03-20 10:40:18 +0000 UTC:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(1): storage call error 0: StandaloneSealCommit: Missing store file (or associated split paths): /mnt/bn01/cache/s-t01197354-1974/sc-02-data-layer-1.dat

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
   1: filecoin_proofs::api::verify_store
   2: storage_proofs_porep::stacked::vanilla::params::Labels<Tree>::verify_stores
   3: filecoin_proofs::api::validate_cache_for_commit
   4: filecoin_proofs_api::seal::seal_commit_phase1_inner
   5: filecoin_proofs_api::seal::seal_commit_phase1
   6: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
   7: ffi_toolkit::catch_panic_response
   8: fil_seal_commit_phase1
   9: _cgo_24921690cad9_Cfunc_fil_seal_commit_phase1
             at ./tmp/go-build/cgo-gcc-prolog:1499:11
  10: runtime.asmcgocall
             at ./usr/local/go/src/runtime/asm_amd64.s:667
13.	2022-03-20 10:41:18 +0000 UTC:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
14.	2022-03-20 10:42:39 +0000 UTC:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(1): storage call error 0: StandaloneSealCommit: Missing store file (or associated split paths): /mnt/bn01/cache/s-t01197354-1974/sc-02-data-layer-1.dat

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
   1: filecoin_proofs::api::verify_store
   2: storage_proofs_porep::stacked::vanilla::params::Labels<Tree>::verify_stores
   3: filecoin_proofs::api::validate_cache_for_commit
   4: filecoin_proofs_api::seal::seal_commit_phase1_inner
   5: filecoin_proofs_api::seal::seal_commit_phase1
   6: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
   7: ffi_toolkit::catch_panic_response
   8: fil_seal_commit_phase1
   9: _cgo_24921690cad9_Cfunc_fil_seal_commit_phase1
             at ./tmp/go-build/cgo-gcc-prolog:1499:11
  10: runtime.asmcgocall
             at ./usr/local/go/src/runtime/asm_amd64.s:667
15.	2022-03-20 10:43:39 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	consecutive compute fails
16.	2022-03-20 10:44:39 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
17.	2022-03-20 10:44:39 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1974} from remote(1): sector not found
18.	2022-03-20 10:45:39 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
19.	2022-03-20 10:45:39 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1974} from remote(1): sector not found
20.	2022-03-20 10:46:39 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
21.	2022-03-20 10:46:42 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1974} from remote(1): sector not found
22.	2022-03-20 10:47:42 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
23.	2022-03-20 10:47:42 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1974} from remote(1): sector not found
24.	2022-03-20 10:48:42 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
25.	2022-03-20 10:48:55 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1974} from remote(1): sector not found
26.	2022-03-20 10:49:55 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}

[...]

	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1974} from remote(1): sector not found
1368.	2022-03-20 23:31:43 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
1369.	2022-03-20 23:31:43 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1974} from remote(1): sector not found
1370.	2022-03-20 23:32:43 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
1371.	2022-03-20 23:32:43 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1974} from remote(1): sector not found
1372.	2022-03-20 23:33:43 +0000 UTC:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
1373.	2022-03-20 23:33:43 +0000 UTC:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {1197354 1974} from remote(1): sector not found

[shotcollin]

Could the sector not found be referring to the file that would have been in the "unsealed" file location? That's the only thing that's definitely no longer there, since it was deleted by the miner at some point.

I have FinalizeEarly set to true in the miner config file. Perhaps it moved the files before the proof was successfully computed, and that included removing the unsealed sector file?

But I also just noticed this value in there: AlwaysKeepUnsealedCopy = true, and that's definitely not happening (and not something I wanted to happen) although now I'm wondering why these files aren't being stored anywhere.

[shotcollin]

Reiterating my main questions here:

1. If both sectors are in the Finalized state, why does only one successfully complete once the miner balance is sufficient? The only difference I can see between the two sectors is that the first one was able to send a commit submit message to the pool before the balance became too low to even do that. But that seems to fail after the second retry, and then it gets SectorCommitFailed with SysErrInsufficientFunds like the second sector. Unlike the second one, the first one continues SectorRetryComputeProof until it's able to complete SectorRetrySubmitCommit. Does this have something to do with the rust/ffi panic detailed in the backtrace?

2. Even more perplexing: once the second sector gets its first SectorCommitFailed error, it goes to SectorRetryComputeProof and fails with SectorComputeProofFailed and a message saying it can't find sc-02-datalayer-1.dat in the cache folder. But this sector was about to be finalized, and sc-02-datalayer-1.dat is a file created during PC1 and deleted by the miner after C2 completes. Is there any situation where a sector in SectorFinalized state would still have sc-02-datalayer-x.dat files stored somewhere? If these files are potentially needed for the compute proof at this stage, why have they already been deleted by the miner? This same thing happened to about 15 other sectors after sector 1974, so it wasn't a one-off thing or a network hiccup or a single missing file. All of the sectors got to SectorFinalized with zero problems, then all failed with the same error about the file not being available.

[shotcollin]

Also posting my "solution"/workaround again here:

Once I discovered this problem and got the miner balance back up, etc., the failed sectors continued on in the same loop. The sectors did finally restart and successfully complete with no additional errors after I did the following:

1. Moved the sector files from their location in long-term storage to somewhere not accessible by the miner. The files in long-term storage were the sealed sector files and the cache directories containing p_aux, t_aux, and sc-02-data-tree-r-last-0.dat through sc-02-data-tree-r-last-7.dat. (I don't think this was actually necessary to get them to complete but I wanted to preserve them just in case.)
2. Created a dummy file for each sector in the miner's sealing storage location in the unsealed directory. (e.g. with touch /sealing-storage/location/unsealed/s-t01197354-1974)
3. Forced an update to the miner's index (or whatever you would call it?) of the seal/store storage locations, e.g. by restarting any worker (or the miner), causing the sector {xxxxxxx yyy} redeclared in zzzzzz log messages.
4. Only after dummy file creation and the sector file redeclaration did sectors restart PC1.