Make UPDATE_HARVESTER_CACHE_INTERVAL accessible in config.yaml

[maxpla3]

The farmer sends a request_plots to the remote harvester every 2 minutes. Here is the log from the farmer:
2021-11-04T07:15:52.229 farmer farmer_server : DEBUG -> request_plots to peer 192.168.1.2 ...
and here is the incoming request in the harvester's log:
2021-11-04T08:15:55.560 harvester harvester_server : DEBUG <- request_plots from peer ... 192.168.1.1

Is it possible to increase the period of these requests from 2 to e.g 30 minutes? I'm asking because these requests seem to trigger some kind of look-ups of the plots on the harvester and since my harvester is very low-performance it frequently leads to seek times way above 5 seconds.

[maxpla3]

There is a variable called UPDATE_HARVESTER_CACHE_INTERVAL which is set to 90. It looks like this is the variable which defines how often the farmer sends request_plots to the harvester. This calls the function request_plots found in harvester_api.py which simply iterates over all plots and gets all kind of information about them. This is what slows down my harvester every 120 seconds or so.

As far as I understand the period UPDATE_HARVESTER_CACHE_INTERVAL can be increased to some larger value without significant drawbacks. The only negative effect will be that the farmer is not up-to-date regarding the amount of plots which is harvested. But the farmer does not have to know that as long as the harvester receives the challenges and returns the proofs if any found. Do I understand it correctly?

Would it be possible to add the variable UPDATE_HARVESTER_CACHE_INTERVAL to config.yaml so that chia users can adjust it as they want?

Edit: There is a commit in which UPDATE_HARVESTER_CACHE_INTERVAL was increased from 60 to 90 seconds "to help large farmers", as it's said in the commit note: d3e726e

[rmica]

Interesting. One question, what are your plots_refresh_parameter settings on the harvester? It's also relevant. I was trying a very lazy setup on a stable harvester with 1k plots, kind of:

  plots_refresh_parameter:
    interval_seconds: 3600 # The interval in seconds to refresh the plot file manager
    retry_invalid_seconds: 1200 # How long to wait before re-trying plots which failed to load
    batch_size: 10 # How many plot files the harvester processes before it waits batch_sleep_milliseconds
    batch_sleep_milliseconds: 1000 # Milliseconds the harvester sleeps between batch processing

[maxpla3]

batch_size: 1000
batch_sleep_milliseconds: 10
interval_seconds: 21600
retry_invalid_seconds: 86400

[rmica]

So you repeat the lookup every 6 hours, don't know the plots count, but a 10ms sleep for each 1k batch, looks too minimal. I'm trying to keep the interval at 3600 in order to be notified of potential drive failures, while running smaller batches with bigger sleeps.

[maxpla3]

That's probably a good strategy which I also considered but have not tested yet. But doing the plot refresh every 6h does the trick for me. It's only that the farmer triggers another "plot checking" every 90 to 120s which I'm trying to fix.

[maxpla3]

I increased UPDATE_HARVESTER_CACHE_INTERVAL from 90 to 1200 and will let the farmer/harvester run over night to see how the system performs.

[rmica]

I'm looking forward to your results.

[maxpla3]

By increasing UPDATE_HARVESTER_CACHE_INTERVAL to 1200 I get much more stable harvesting. With the default value I would get delays in responses to the incoming challenges of up to 30 seconds which happened about every 120 seconds. Now this kind of delay happens only every 20 minutes due to the increased UPDATE_HARVESTER_CACHE_INTERVAL. I don't see any negative effects.

I have about 7800 plots on over a hundred external HDDs and with the modified UPDATE_HARVESTER_CACHE_INTERVAL and config.yaml the harvester runs totally stable with seek times mostly between 1.5 and 2.5 seconds.

[rmica]

Ok, so it might be set even higher.

[loppefaaret]

there is an option in config.yaml for this, under the harvester: section:

  plots_refresh_parameter:
    batch_size: 100
    batch_sleep_milliseconds: 1000
    interval_seconds: 3600
    retry_invalid_seconds: 1200

you can check the initial config here for newly added options:
https://github.com/Chia-Network/chia-blockchain/blob/main/chia/util/initial-config.yaml

[maxpla3]

You got it wrong. I know that one can adjust these parameters for the harvester. I am proposing to make UPDATE_HARVESTER_CACHE_INTERVAL from farmer.py adjustable via config.yaml. This is not the case yet.

[loppefaaret]

fair enough, ill note it down here

[loppefaaret]

the remote harvester you see this effect on, is holding roughly 7k plots ?

[maxpla3]

Yes, about 7800 plots. It takes some time to iterate over all of them and determine their file size, date of last modification and what not. I don't see why this should happen every 90 seconds on a farming setup where no new plots are added.