can't pickle module objects

[minwang-ai]

Hi,

What’s the meaning of can not pick pickle module object for trainer?
Same errors for trainer.fit()

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/fs/applications/pycharm-ce/2019.1.1/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/fs/applications/pycharm-ce/2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/mwa7rng/mwa7rng/run_repos1/FFB6D/ffb6d/main.py", line 167, in <module>
    main()
  File "/home/mwa7rng/mwa7rng/run_repos1/FFB6D/ffb6d/main.py", line 141, in main
    lr_finder = http://trainer.tuner.lr_find(model, datamodule=dm, min_lr=1e-08, max_lr=1, mode="linear", num_training=100)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/tuner/tuning.py", line 197, in lr_find
    "update_attr": update_attr,
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1094, in tune
    result = self.tuner._tune(model, scale_batch_size_kwargs=scale_batch_size_kwargs, lr_find_kwargs=lr_find_kwargs)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/tuner/tuning.py", line 53, in _tune
    result["lr_find"] = lr_find(self.trainer, model, **lr_find_kwargs)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/tuner/lr_finder.py", line 238, in lr_find
    trainer.tuner._run(model)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/tuner/tuning.py", line 63, in _run
    self.trainer._run(*args, **kwargs)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1195, in _run
    self._dispatch()
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1274, in _dispatch
    self.training_type_plugin.start_training(self)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
    self._results = trainer.run_stage()
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1284, in run_stage
    return self._run_train()
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1306, in _run_train
    self._run_sanity_check(self.lightning_module)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1366, in _run_sanity_check
    self._evaluation_loop._reload_evaluation_dataloaders()
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 169, in _reload_evaluation_dataloaders
    self.trainer.reset_val_dataloader()
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py", line 521, in reset_val_dataloader
    RunningStage.VALIDATING, model=pl_module
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py", line 436, in _reset_eval_dataloader
    dataloaders = [deepcopy(train_dataloader) for _ in range(len(dataloaders))]
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py", line 436, in <listcomp>
    dataloaders = [deepcopy(train_dataloader) for _ in range(len(dataloaders))]
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle module objects

[rohitgr7]

specifically for this part of the codebase, it will be resolved by: #9989

[tchaton]

Dear @MinWang1997,

From the trace, it seems your DataLoaders can't be deepcopied.

dataloaders = [deepcopy(train_dataloader) for _ in range(len(dataloaders))]

Be mindful you can't deecopy local objects or lambda function.

Furthermore, the more details you provide, the simpler it is for us to help you. A reproducible script is the best way to get you to debug your problems.