[AMP] Full Precision in Validation

[hoangtnm]

I want to use AMP to train GANs in mixed precision to accelerate training time, however, in validation_step and validation_epoch_end, the model seems to run in mixed precision rather than full precision. Therefore, I cannot inference in full precision to visualize it's image generation output. Is there any way to run in full precision at validation when using precision 16?

[justusschock]

Hi,

I'm afraid there isn't (at least not that easily). The only way I could think of is you implementing your own PrecisionPlugin.

That being said, I also cannot think for a reason why you want to run validation in full precision when training in mixed.

The operations to run should be the same and AMP only converts "safe" operations to 16bit.

[hoangtnm]

@justusschock because I use GANs for generating images, therefore, I think it is necessary to generate images from a validation set after every epoch to check whether it is generating good images or not, which means the GAN model should be run in full precision (disable auto_cast in AMP only in validation phase) to generate images properly.

[justusschock]

AMP should not have any effect for image generation. It should run fine with and without AMP.

[mactavish91]

@justusschock Hello, bro, if I want to conduct mixed accuracy training with little impact on accuracy, which parameter should I use? precision=32, amp_backend="native" or precision=16, amp_backend="native"？