Use cpu affinity to determine number of workers in blockchain.py (cpu memory usage reduction and Sync speed increase) #8441
Closed
grayfallstown
started this conversation in
Ideas
Replies: 1 comment
-
if you submit it to our repo, it might have a chance of catching the dev's attention ? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
This feature request aims to improve sync speed and reduce resource consumption while co-farming chia and its forks.
Currently, blockchain.py uses
os.cpu_count() - 2
to determine the number of workers it should start, see blockchain.py. This ignores the cpu affinity, as in if the process is limited to 4 cores, it will still start 14 workers on an 8 core, 16 thread cpu, all of them running on just 4 logical cores.This is a nuisance when only chia itself is running, but becomes a huge problem if you co-farm chia forks as well, as each worker uses about 50 MB of ram and quite a bit of cpu time. With 40+ instances (chia + a lot of the forks) this uses 27+GB of ram for workers alone and slows down syncing a lot, as the 560+ workers fight over 16 logical cores. Using cpu affinity would allow users to limit the number of workers and distribute the cpu load.
I understand that pythons
os.sched_getaffinity
is only available on some unix platforms, but windows implements the necessary api as well.One library that implements this platform independent is affinity. I understand, that adding external libraries needs to be well considered. This library is small and therefore relatively quick to review. Alternatively, the necessary api call could be added directly in the chia code. Sadly, my python experience is not sufficient to be confident over making that pull request myself.
An alternative:
A parameter could be added to the config.yaml that allows the user to limit the number of workers to the configured value, if the user chooses to set that parameter. This has the disadvantage of not being able to restrict the worker to certain cpu cores. The code necessary would be available in the covid chia fork repository (it is my pull request, feel free to merge or tell me that I should make that pull request for chia as well)
Beta Was this translation helpful? Give feedback.
All reactions