boring model for DDP

[hkmztrk]

Hello,

We are (@haimasree) trying to run the Boring Model example in DDP setting on LSF cluster to test our configuration and have the following issue:

 ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'test_run.<locals>.TestModel'

Is it possible to have a boring model example that could run on DDP? Also it'd be great if you could point out if there is a special configuration needed for LSF different than SLURM.

• CUDA:
  GPU: (one of the randomly assigned ones)
  - Quadro RTX 6000
  - Quadro RTX 6000
  - Quadro RTX 6000
  - Quadro RTX 6000
  • available: True
  • version: 10.2
• Packages:
  • numpy: 1.17.0
  • pyTorch_debug: True
  • pyTorch_version: 1.7.0
  • pytorch-lightning: 1.2.1
  • tqdm: 4.58.0
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.7.10
  • version: Proposal for help #1 SMP

@williamFalcon @Borda @tchaton @awaelchli

[awaelchli]

I assume you are running with ddp_spawn, so that means you need to move the definition of the model class (TestModel) outside to the main file, like so:

...

# move class here:
class TestModel(BoringModel):

        def on_train_epoch_start(self) -> None:
            print('override any method to prove your bug')


def test_run():
    # remove definition from here and move outside of function

... 

There is a LSF PR here #5102

[hkmztrk]

Thanks @awaelchli ! No, we were actually running with 'ddp', never tried 'ddp_spawn'. Would it make a difference in your above suggestion?

[awaelchli]

We typically see this when using ddp_spawn which requires everything to be pickleable, and because you had a pickle error I thought that it is the reason for it.

But it means something else is trying to pickle the process, maybe a checkpoint saving operation?
Unfortunately only the 2 last lines of the error message were posted. If we had the full stack trace we would know.

[haimasree]

Thanks a lot for your prompt reply. Here is the full traceback

/home/h879a/.conda/envs/glmtest/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:50: UserWarning: You requested multiple GPUs but did not specify a backend, e.g. `Trainer(accelerator="dp"|"ddp"|"ddp2")`. Setting `accelerator="ddp_s$
  warnings.warn(*args, **kwargs)
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
Traceback (most recent call last):
  File "example.py", line 152, in <module>
    test_run()
  File "example.py", line 146, in test_run
    trainer.fit(model, train_data, val_data)
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 513, in fit
    self.dispatch()
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in dispatch
    self.accelerator.start_training(self)
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 74, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp_spawn.py", line 106, in start_training
    mp.spawn(self.new_process, **self.mp_spawn_kwargs)
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 148, in start_processes
    process.start()
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/h879a/.conda/envs/glmtest/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'test_run.<locals>.TestModel'

[awaelchli]

We can see it is using mp.spawn(self.new_process, **self.mp_spawn_kwargs), that explains why you get the error and my fix will work. However, since you mention you used accelerator="ddp", there must be a bug in the accelerator connector that selects ddp_spawn anyway.

What are your trainer arguments? And have you set any special environment variables?

[haimasree]

Thank you for your continued support.

Trainer arguments are the following:
gpus: 4
max_epochs: 4
accelerator: 'ddp'
profiler: 'True'

I set the following environment variables
export NCCL_DEBUG=INFO and append PYTHONPATH to some locally installed package path