[BUG] Challenge times increase during final copy of plots

[Akaino]

Might not be an actual bug
Challenges take longer when new plots are being copied to any plot directory.

To Reproduce
Plot something. Wait for final copy. Check challenge times.

Expected behavior
Challenge times shouldn't take longer, I guess?

Screenshots
Challenges during copy

Plot 1/2 copied successfully

Plot 2/2 copied successfully

Desktop (please complete the following information):

• OS: Windows 10 [10.0.19041]
• OS Version/Flavor: Windows 10 [10.0.19041]
• CPU: Intel Core i7 9900k

Additional context
The application simply reads logs from the default logs directory and throws data into the graph and table.

Plotting with cli, 2 parallel, staggered by 60 minutes:

chia plots create -k 32 -b 4000 -r 4 -u 128 -n 28 -t T:\Temp -2 T:\Temp2 -d Q:\

T: = TempDir [NVME]
Q; = FinalDir [SATA HDD] or [Synology NAS] (issue exists in both cases).

So far I'm not missing challenges.

[sargonas]

This is likely due to the fact that the challenge time is a result of system resource availability, especially drive I/O . If you are peaking some of your I/O on the drive you’re farming from when trying to copy the files into it, then the challenges are going to take a few milliseconds longer while the copy is underway as the threads wait their turn.

[Akaino]

I'll give it a shot and drop the plottings targeting the NAS as final.

[Akaino]

Ok, I figured copying to the NAS indeed does increase challenge times a lot.
I'm constantly getting challenges >20s with a lot >30s. Even some 400-500s!!
However, I don't get why that happens. I understand that a NAS might not be the perfect storage considering network traffic etc. But copying to it really shouldn't affect challenge times by that amount.
I'm going to try limiting bandwidth to the NAS to see whether it's enough to keep challenge times in check.

Edit:
I tried the following now:
Limiting Incoming Newtork Bandwidth on the NAS to ~0.5GBit. No difference.
Limiting Data Transfer with Windows QoS Group Policy. Policy was ignored (Did I say I hate Windows?).
Limiting SMB Data Transfer with PowerShell New-NetQosPolicy -Name "SMBRestrictFileCopySpeed" -SMB -ThrottleRateActionBitsPerSecond 500MB. Transfer was slower now. Challenge times still skyrocketed.

I think I'll try setting up a Pi4 being Full Node and Farmer? I can't see why that would help me with the challenge times if it's the NAS' issue though.

[sargonas]

Oof... glad to hear it's narrowed down, but bummer it is indeed your NAS. :/ (I, too, hate Windows in scenarios like this... "oh hey I know you set this setting but here's the thing, I have 3 different ways to toggle something from 7 different functions due to 8 layers of legacy code andUIs sooooooo nah I don't think so bud.")

[Akaino]

I don't quite get the issue here.
I see how it takes longer to check the plots. However, as long as there are 0 plots passed, everything is quite under 30s.
Once a plot passes, the check takes ages. This can't be the NASs issue alone.

This is now during robocopy moving the files from SSD to NAS...

16:30:51.669 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 94.62579 s. Total 180 plots
16:32:11.037 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 133.66629 s. Total 180 plots
16:33:57.899 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 135.03200 s. Total 180 plots
16:33:57.902 harvester chia.harvester.harvester: INFO     2 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 164.53450 s. Total 180 plots
16:33:57.933 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 106.88750 s. Total 180 plots

16:27:31.186 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.42550 s. Total 180 plots
16:27:44.845 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.65800 s. Total 180 plots
16:28:11.816 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.44450 s. Total 180 plots
16:28:53.457 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.48800 s. Total 180 plots
16:29:01.432 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 6.94750 s. Total 180 plots
16:29:17.042 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 10.99400 s. Total 180 plots
16:29:57.370 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.47700 s. Total 180 plots
16:30:24.285 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.54300 s. Total 180 plots
16:30:37.994 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.70800 s. Total 180 plots
16:30:51.660 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.66479 s. Total 180 plots
16:31:06.263 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.57800 s. Total 180 plots
16:31:13.366 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 7.10250 s. Total 180 plots
16:31:29.376 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 11.35900 s. Total 180 plots
16:31:42.866 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.48850 s. Total 180 plots
16:32:10.018 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.53600 s. Total 180 plots
16:32:38.371 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.66400 s. Total 180 plots
16:33:19.482 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 13.52550 s. Total 180 plots
16:33:27.658 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 8.17450 s. Total 180 plots
16:33:32.255 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 4.59650 s. Total 180 plots
16:33:43.357 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming b3a98ece13... Found 0 proofs. Time: 11.10100 s. Total 180 plots

[Akaino]

For those interested, I mitigated my issue using robocopy with it's /ipg parameter.
This parameter depends on your network connection so I can't really tell you what's best for you... I'm going with this:
robocopy $source_folder $destination_folder_NAS /MOV *.plot /IPG:10
Works pretty well so far. No issues.

[nomadengineer]

It could be down to Random seek times during a file copy?

[Akaino]

I think it was just Windows being Windows... Copy processes seem to be considered highest priority. Everything else went down for the entire duration.
I mitigated it with a robocopy delay though (ipg:10 in my case).
robocopy $source_folder $destination_folder_NAS /MOV *.plot /IPG:10

[nomadengineer]

I would like to understand this in a bit more detail please.
So when the chia daemon is copying the files from tmpplot -> farm directory, windows being windows will always cause these times to jump?
So we keep the plots in another folder on the tmp drive and use robocopy to send to the farming destination?

I hope I got it!
Thanks.

[milan-patartics]

I think challenge processes should run as the highest priority processes by default.

[BrandtH22]

If you are still experiencing the issues it is recommended to upgrade to the latest version of chia. Please download it from here: https://www.chia.net/downloads/

Since this issue has been open for a number of years without additional comments, we will be closing this ticket but if we have closed this ticket in error do not hesitate to reach out to us again with any followup questions, comments, or if the issue persists after an update.

Please note that upgrading major versions of chia generally requires the config.yaml to be recreated using these steps:
1. Stop chia
2. Rename the config or move it to a different directory
3. In CLI run chia init
4. Copy any custom settings from the old config to the new (plot drive directories, pool info, trusted node info, etc) - be careful here as yaml is very picky about formatting including those leading spaces
5. Start Chia

The best place to reach our support team is on Discord (https://discord.gg/chia) or by reopening this ticket.