"resume from checkpoint" lead to CUDA out of memory

[Defiler24]

When I use “resume from checkpoint”,
there is a “CUDA out of memory” problem,
when using torch.load(), set "map location" to "cpu" can solve this problem,
in "resume from checkpoint" scenario, what should I do?

[four4fish]

@Defiler24 Could you share which training strategy(plugin) are you using? Or could you share your code here

[Defiler24]

I did not set a specific strategy, the default should be DDP spawn.

data  = DWheel(**vars(args))
model = MWheel(**vars(args))

trainer = Trainer(
                max_epochs=480, 
                gpus=8, 
                sync_batchnorm=True,
                resume_from_checkpoint="../version0/checkpoint/epoch=150.ckpt")
trainer.fit(model, data)

[Defiler24]

I solved the problem after setting the strategy to 'ddp'.

[YueyangLiulyy]

I have exactly the same issue, the only difference is that my model is trained on single GPU. I did not specify any stratgey as well, is there any solution to solve it? Thanks