model weight file corrupted when training on multi-gpus

[kleinzcy]

When I trained on multi-GPU, the model weight file is corrupted. I guess the reason is that multi-gpus are saving the model weight at the same time. So how can I call the CheckpointCallback on a specific GPU?

[rohitgr7]

checkpoints are saved only on global rank 0 which means only one of the GPUs will save the checkpoint even in the case of multi-node training. The issue might be something else.