AMP: Native AMP doesn't work in my Gigabyte Aorus Master RTX 3070 environment

[roma-glushko]

I'm trying to use AMP in my new Gigabyte Aorus Master RTX 3070 environment.
I have changed precision config from 32 to 16 and back to track global memory consumptions. Surprisingly, in both cases I see the same memory consumption (batch_size was the same).

Though, during training initialization, PyTorch Lightning seems to recognise that AMP should be used for the training:

Despite that, memory consumptions are the same:

I expect AMP to reduce my memory consumptions.

My dependencies:

• PyTorch was installed this way:

pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 -f https://download.pytorch.org/whl/torch_stable.htm

• PyTorch Lightning: 1.2.3
• Python: 3.8.6

Any clues why it does work for me?

[roma-glushko]

Hey @Borda @williamFalcon 👋 have you seen anything similar to this issue?

[roma-glushko]

I had a conversation with PyTorch community about this issue to exclude a case of environment misconfiguration on my side (https://discuss.pytorch.org/t/rtx-3070-amp-doesnt-seem-to-be-working/118835/7)

We came up to a conclusion that, generally, AMP works there, but there is a project, that I originally mentioned, which possibly has a combination of model/dataset that triggers an issue on Lightning side.

[awaelchli]

if you can share a small code sample that shows the problem on the latest Lightning version, ideally using our debugging template, and post it as a bug report in github then we can look into the issue.