Using GPU and CPU simultaneously for parameters

[ssharpe42]

How do you move parameters to the CPU while training on the GPU without any example tensors for .type_as?

Would you just make a global cpu tensor to refer to in this case?

[rohitgr7]

wouldn't just .to('cpu') work in this case?

[ssharpe42]

Ok thought .to() was just discouraged in PL

[rohitgr7]

it is but only when you create new tensors since during distributed training calling new_tensor.cuda() won't be correct all the time. Although you can still use .to calls and access CPU using cpu and device using self.device.