Switch multiprocessing to concurrent.futures #3
Description
Hi there, thanks a lot for making this!
I've recently run into an issue where pyrho make_table ran into a state where it didn't fail, but also stopped using any CPU. It turned out that one of the worker processes that are being spawned was killed because it exceeded the memory limit (I'm running on a slurm managed cluster that enforces set memory limits), triggering this bug in python. This ended up being fixed in concurrent.futures.ProcessPoolExecutor, however, the PR fixing it in multiprocessing, while getting some attention recently, is unclear if it will be merged into a python version soon..
Given the straightforward usage of multiprocessing in this repository, switching to concurrent.futurures.ProcessPoolExecutor should be fairly simple and would remove the possibility of some frustrations debugging it from programs that use this, which are probably often run in environments in which this can happen.
I'll create a PR closing this immediately.
Cheers!
Georg