Harvester latency increase over time

[vladaepro]

I'm struggling to find out why my harvester response time is about 0.2 - 0.3 seconds in first few hours after i start chia app, and then just increase to 2-3 seconds, and continue even further. If i close the app and start again, everything is fast again (for next few hours).

details about pc: i7 8700, 24gb ram, 4 sata + 18 external HDDs (about 2100 plots).
It's dedicated harvesting machine, no plotting, copying or anything else on it.
I'm using latest (1.2.5) app

What i've tried until now (without success)

• cool down hdds (all are below 40C now)
• rearrange usb ports and hubs
• uninstall AV and all unnecessary apps from PC
• disable power management (sleep) for every usb device/hdd
• benchmarked every single hdd to confirm all of them are working at decent speed (~100 MB/s average)

I would understand that there's some hardware/software issue if i'm having big latency all the time, but can't understand why just after hours of harvesting.

P.S. i just noticed that start_harvester process CPU usage become quite high (15-17% on 8 threads i7 CPU) when latencies start rising. On the beginning, it's low, up to 2%.

here's how my heatmap usually looks like (green 'field' last about 4-5 hours)

[loppefaaret]

1.2.5 introduces a new cache on the harvester, and as it builds up, it might be pulling the perfomance down...
thank you for the report, and i think ill try and poke a dev about this, just to make sure that we gt everything going as best as possible for all users

[vladaepro]

Thanks for answering. Just to add that i found (quite ugly) workaround, maybe that info means something to devs - if i run plots check, and let it just gather all plots (i break it when checking is started, no need for that), latency is good again, so as CPU usage.

Based on this, one would assume that hard drives are going to sleep, but i disabled sleep on every possible place in device manager and control panel, and in addition i let an app running which write and delete file to every hard disk periodically, so i don't think they are going to sleep.

However, if anyone knows how to track does they go to sleep or no, please let me know.

[loppefaaret]

18 externals - many externals actually ignores any power schemes or commands from an OS, and force themself to either sleep, or in some cases, lowering their spinrate - but i fail to see how it would only sohw itself after 4~hours - it would be visual in the first hour...
if you feel adventurous - you could try downgrading to 1.2.3 - and see if 8 hours of harvesting using that shows a different heatmap - perhaps you already got that info ?

[vladaepro]

well, i can't claim it was exactly 4 hours, maybe even 1-2hr. however, i made small script which runs on 10 minutes and read random data from disks, so i'll see will it work

[vladaepro]

Just to confirm - my external HDD-s have to be used almost constantly to prevent them going to sleep (or some other 'slow-response' state), despite all settings adjusted in Windows. Making a script which periodically write small data to every hdd didn't solved the issue completely, because it seems that the smaller data you access, the quicker it will go to sleep. So finally i solve it by script which read every plot on every hdd at random offset 0-100gb. That's enough for about 5 minute of low latency, but fortunately whole script execution time is not big, and it doesn't affect current farming. Now my average latency is about 0.4s.
If someone has similar issue i will gladly share the script.

[BrandtH22]

@vladaepro , thank you for the update and confirmation that you were able to resolve the issue. Since this ticket was opened a number of years ago without further comments it seems this issue has been resolved.
We will be closing this ticket but if we have closed this ticket in error do not hesitate to reach out to us again with any followup questions or comments.

The best place to reach our support team is on Discord (https://discord.gg/chia) or by reopening this ticket.