Error when implementing Mixed Precision (16-bit) Training

[oschan77]

Hello there,
I have trouble implementing Mixed Precision (16-bit) Training. I am trying to use precision=16 in Trainer to reduce my memory used in GPUs.
trainer = pl.Trainer(profiler='pytorch', max_epochs=1000, num_nodes=1, gpus=8, logger=tb_logger, strategy='ddp', precision=16) trainer.fit(stylegan2_model)

However, I encountered this error.
File "/scratch/PI/cemclo/stylegan2ada_pl.py", line 193, in forward out = F.linear(x, self.weight(), bias=self.bias) RuntimeError: expected scalar type Float but found Half

What causes this issue and how could I solve this issue? Thanks a lot.

[akihironitta]

Hi @oschan77! Could you share your script or reproduce it with the BoringModel? https://github.com/PyTorchLightning/pytorch-lightning/tree/master/pl_examples/bug_report

[m13uz]

link broken

[akihironitta]

https://github.com/Lightning-AI/lightning/tree/0ca3b5aa1b16667cc2d006c3833f4953b5706e72/examples/pl_bug_report