Multiprocessing is broken: `TypeError: cannot pickle '_thread.RLock' object`

[petargyurov]

Bug

When trying to run detections with n_cores > 1 and disabling GPU, the runner uses a multiprocessing approach. Currently there is a bug that produces the error: TypeError: cannot pickle '_thread.RLock' object

Traceback

Traceback (most recent call last):
  File "C:/Users/pgyur/Documents/My Projects/megadetector-api/main.py", line 6, in <module>
    results = tf_detector.run_detection(input_path='test_imgs/animals',
  File "C:\Users\pgyur\Documents\My Projects\megadetector-api\tf_detector.py", line 121, in run_detection
    results = pool.map(self.__process_images, image_batches)
  File "C:\Program Files (x86)\Python38-32\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Program Files (x86)\Python38-32\lib\multiprocessing\pool.py", line 768, in get
    raise self._value
  File "C:\Program Files (x86)\Python38-32\lib\multiprocessing\pool.py", line 537, in _handle_tasks
    put(task)
  File "C:\Program Files (x86)\Python38-32\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Program Files (x86)\Python38-32\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object

Details

From a quick search it seems that it may be related to a new behaviour introduced somewhere in Python 3.8.0 or above, but not sure.

Workaround

Using multithreading with ThreadPool insted of multiprocessing with Pool, appears to resolve the issue but I am not sure if that is a good approach.