TypeError: cannot pickle '_thread.RLock' object

[LanceKnight]

Hello,

I'm trying to write a scheduler (using startmap from multiprocessing) so I can run multiple pytorch_lightning jobs at the same time. I worked fine when I used just 1 gpu. But when I changed --gpus from 1 to 2. I got the following error. Can someone help?

Traceback (most recent call last):
File "/home/live-lab/projects/unified_framework/experiments/exp153_full_param1/kgnn/entry.py", line 345, in
main(gnn_type, use_clearml)
File "/home/live-lab/projects/unified_framework/experiments/exp153_full_param1/kgnn/entry.py", line 308, in main
actual_training(model, actual_training_data_module, use_clearml,
File "/home/live-lab/projects/unified_framework/experiments/exp153_full_param1/kgnn/entry.py", line 261, in actual_training
trainer.fit(model=model, datamodule=data_module)
File "/home/live-lab/environments/venv/kgnn-env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit
self._call_and_handle_interrupt(
File "/home/live-lab/environments/venv/kgnn-env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 719, in _call_and_handle_interrupt
return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
File "/home/live-lab/environments/venv/kgnn-env/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/spawn.py", line 78, in launch
mp.spawn(
File "/home/live-lab/environments/venv/kgnn-env/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/live-lab/environments/venv/kgnn-env/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 189, in start_processes
process.start()
File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/lib/python3.10/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/usr/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object

pytorch lightning version: 1.6.3
python version: 3.10.4
pytorch version: 1.11.0+cu113

[akihironitta]

@LanceKnight It's a limitation of spawn-based strategies such as ddp spawn. I believe you won't see the error if you use ddp instead.

ref: https://pytorch-lightning.readthedocs.io/en/stable/accelerators/gpu.html#pickle-errors