save checkpoint issue in nested trainer

[eddiecong]

Hi all, I use multi-GPU to train the model, this error happens when saving the checkpoint. But using a single GPU to train the model, everything works fine.

TypeError: cannot pickle '_thread.lock' object

I wonder if it is because I nest the trainer inside another model, the codes for a nested trainer is as followed:

self.model.trainer = self.trainer

So many thanks in advance.

[ethanwharris]

Hi @eddiecong, the trainer is not pickleable and so you're probably correct that it's to do with your nested trainer code. You could override the on_save_checkpoint hook and then delete the trainer from the checkpoint as suggested by this answer: #3444 (comment)

Hope that helps 😃

[eddiecong]

Hi @ethanwharris , thanks so much. This helps me a lot!

[GuillaumeTong]

This is actually not completely sufficient if you intend to run trainer.validate/test/predict after training, since trainer._run will attempt to do parsing.clean_namespace(model.hparams). I can see you have a is_picklable function in there, but it fails to catch TypeError.