Error while saving model

[nickKyr]

Hello pytorch lightning community,

I get a strange error while saving the model via pl.callbacks.ModelCheckpoint when I use multiple GPUs (dp accelerator). Based on an online search, I should set pickle version to 4? I know that in torch.save there is a pickle parameter, but is there something similar in the Trainer API or in ModelCheckpoint?

I will try to reproduce the error in the boring model. You can see below the error I get.
Thanks

  File "/python3.7/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 404, in save_checkpoint
    atomic_save(checkpoint, filepath)
  File "/python3.7/site-packages/pytorch_lightning/utilities/cloud_io.py", line 63, in atomic_save
    torch.save(checkpoint, bytesbuffer)
  File "/python3.7/site-packages/torch/serialization.py", line 372, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol)
  File "/python3.7/site-packages/torch/serialization.py", line 476, in _save
    pickler.dump(obj)OverflowError: cannot serialize a string larger than 4GiB
OverflowError: cannot serialize a string larger than 4GiB

[awaelchli]

Something in your checkpoint is larger than 4GB, could you figure out what?
You can implement the on_save_checkpoint hook in LightningModule to inspect it, if that helps.

To answer your question, there is currently no way to change the pickle parameter in torch.save, but not sure you should do that or even that this will solve your problem.