Bad file descriptor (os error 9) causing window post to fail for provable sectors, reproduced on multiple systems running miner #7961
Answered
by
Reiers
shotcollin
asked this question in
Storage Provider
-
Lotus VersionDescribe the BugHello, I'm having a miner problem where at the beginning of the proving period it reports all the sectors are good, but then immediately starts throwing errors. The errors include the text "Bad file descriptor (os error 9)" and repeat for the remainder of the proving period. The errors stop after that, and the miner appears to be operating fine for the rest of the day. The sector files are there, which is confirmed by the faults being declared resolved before each day's deadline, as well as by several successful runs of The issue started around the 8th or 9th of Jan, but I'm not sure why. I think that's around the time I did a small network reconfiguration, but nothing on the Lotus machines changed during that time, and I don't think I've changed anything else with them since then. Between the 8th and today I've been able to successfully prove at least one round of deadlines. I consulted with @f8-ptrk and @LexLuthr who suggested the errors may indicate a hardware problem, and after moving the miner to a different machine using the backup/restore function, the next deadline was successful. But then the following day the same thing started happening again on the new miner. I also fully rebuilt Lotus on the new miner and checked for missing dependencies. The new miner reported some mismatched parameter cache file checksums (I'm not sure if the files were corrupted or if the parameters changed?) but I was able to download correct files before the proving deadline started, and still got the errors. FD limit on the miners is 1048576. I also attempted to increase it in the service file after seeing some related issues, but that didn't have any effect on the problem. Here's the service config section for my miner. It's basically the same on each machine I've tried. The
Below are the errors that repeat throughout the proving period. I'm also attaching a combined rust/go log with debugging on that covers the last few days. On one or two of the days I just shut everything down right after the errors began reocurring so that's why there's a break at that point on some day(s). Logging InformationRepo Steps
|
Beta Was this translation helpful? Give feedback.
Answered by
Reiers
Jan 16, 2022
Replies: 1 comment 8 replies
-
|
Hi @shotcollin Sorry to hear that - thats a lot of slashing.
This will check the params, if they are already downloaded. And you should be able to get the
Im not able to re-produce the issue. So this is not Lotus software related, this is hardware malfunction,network or incorrect configs, permissions etc. I have only encounter this in the past and it turn out to be broken hardware every time. Moving this issue to Lotus Discussions for help and troubleshooting. |
Beta Was this translation helpful? Give feedback.
8 replies
-
|
To be safe I did the ram-removal test on the machine that's running the miner and the one that had the share where params were stored. Before testing i downloaded new params onto the miner machine and stored them locally. I just finished the tests and, surprisingly, neither system returned errors for any of the 8 lotus-miner fetch-params 32G I ran on each after taking out 1 stick of ram. Waiting to see what will happen when deadline 0 rolls around here in a few minutes… |
Beta Was this translation helpful? Give feedback.
-
|
@shotcollin Hey, just a quick question, maybe you have already answered it above. Have you moved the params to a local disk rather than a shared one? If not then can you run this test as well? Move the params to local disks for the miner and workers. Bad file descriptors are bad pointers, since currently we are pointing over the network, I just want to eliminate it as a possibility. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @LexLuthr actually I was just about to let @Reiers know that my miner just successfully submitted window post (!!). Given that none of the ram seems to have been faulty based on the test, I'd guess it was the storage? Because I did create a local copy of the param files and used that one today. I can't remember why I was using a shared location in the first place, although it may have been to test whether reading from a raid 0 array sped up the param file check. But I think what might have happened is the old miner created or corrupted a file recently, because when I switched machines it did identify checksum mismatches. Just not sure why it didn't fix it the first time I downloaded new files. In any case, seems like it was definitely hardware-related as you both suggested, and most likely the storage. Anyway thanks a bunch for your assistance, I wish I would have asked about this from the get go instead of fiddling around myself for a few days. |
Beta Was this translation helpful? Give feedback.
-
|
That's great news! That was some serious dedicated debugging. Keep observing for few days and let us know how it goes. |
Beta Was this translation helpful? Give feedback.
-
|
This reminded me of a couple related questions I have. Happy to create a new discussion post if that's preferable, just let me know.
Sorry for the super long second question. Hopefully not too much to slog through. Also the second window post just went though without a hitch! Thanks again for your help, it's much appreciated. |
Beta Was this translation helpful? Give feedback.
Answer selected by
shotcollin
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi @shotcollin
Sorry to hear that - thats a lot of slashing.
I usually take go 1 by 1 stick, boot up the machine
This will check the params, if they are already downloaded. And you should be able to get the
mismatched parameter cache file checksumsERROR/WARN just by doing that. When you get error, you have found the broken RAM…