SWA callback blows up memory requirement during the batchnorm update run

[harryle95]

I was using the previous version of lightning and SWA worked fine. However, when I updated to the latest version of lightning, the batchnorm forward pass run exceeds my GPU memory cap. I suspect this is due to gradient accumulation (I did not use any) in the source code. Can anyone offer any suggestion to address this, I can't revert to prev version due to dependencies.

[grudloff]

There is issue #9441 regarding this problem