Lightning module property self.precision is not consistent

[karthi0804]

I am using pl version 1.3.0. I found that self.precision of pl_module outputs 32 when the precision set to 32 in trainer but outputs 'mixed' when the precision is set to 16 in trainer. Curious as to know why not 16?

[ananthsub]

@justusschock @awaelchli this looks like a real issue. from the mixed precision plugin: https://github.com/PyTorchLightning/pytorch-lightning/blob/28103c67c2baa5810d8e76f77176361e3f61076b/pytorch_lightning/plugins/precision/mixed.py#L27

https://github.com/PyTorchLightning/pytorch-lightning/blob/28103c67c2baa5810d8e76f77176361e3f61076b/pytorch_lightning/accelerators/accelerator.py#L404-L406

https://github.com/PyTorchLightning/pytorch-lightning/blob/28103c67c2baa5810d8e76f77176361e3f61076b/pytorch_lightning/trainer/properties.py#L396-L398

https://github.com/PyTorchLightning/pytorch-lightning/blob/28103c67c2baa5810d8e76f77176361e3f61076b/pytorch_lightning/trainer/connectors/model_connector.py#L30

https://github.com/PyTorchLightning/pytorch-lightning/blob/28103c67c2baa5810d8e76f77176361e3f61076b/pytorch_lightning/core/lightning.py#L97-L98

why didn't mypy flag this?

@shuyingsunshine21 this might lead to underperformance in fp16 for sharded ddp because of this check: https://github.com/PyTorchLightning/pytorch-lightning/blob/28103c67c2baa5810d8e76f77176361e3f61076b/pytorch_lightning/plugins/training_type/sharded.py#L57

[karthi0804]

@ananthsub is it why there is no perf improvement when I use fp16 and profile my training? I can see a slight reduction in the memory usage but no improvement in speed. I tried to increase batch size then also I find no improvement with fp16. I am not if there is a bottleneck in my model causing this prob.

[shuyingsunshine21]

@ananthsub and @karthi0804 , good catch, did not expect that trainer.precision diverges.

For ShardedDDP, we could change

https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/plugins/precision/sharded_native_amp.py#L30

to make precision attribute to 16

In general, I am wondering, what is difference between mixed and 16, should we unify the terminology, for deepspeed, looks like it is integer (16 or 32.)
https://github.com/PyTorchLightning/pytorch-lightning/blob/28103c67c2baa5810d8e76f77176361e3f61076b/pytorch_lightning/plugins/precision/deepspeed_precision.py#L34

[awaelchli]

why didn't mypy flag this?

why should it? it's not connected statically and can only be determined at runtime afaik.

self.precision of pl_module outputs 32 when the precision set to 32 in trainer but outputs 'mixed' when the precision is set to 16 in trainer.

Trainer(precision=16) does not guarantee you 16 bit precision for everything, that's just a wrong advertising as you see it in the TV ^^ When we did the precision plugins we decided that we want to structure the plugins and the naming based on their capabilities and not just by binary classificatin 32 bit or 16 bit, because it's just misleading.

is_fp16 = self.lightning_module.trainer.precision == 16

What would that check have to look like for a user custom plugin? what does is_fp16 refer to here and what do you need it to be in sharded? that the weights must be in half precision?

And same question, what does 16 bit mean in deepspeed?

[awaelchli]

We can think of adding granular properties that determine which elements of the model and data are managed in a particular tensor precision by that plugin, is that something that would help you here?

[shuyingsunshine21]

Trainer(precision=16) does not guarantee you 16 bit precision for everything, that's just a wrong advertising as you see it in the TV ^^

Agree, yeah. So wondering whether makes sense to enforce user to explicitly pass Trainer(precision='mixed') or at least warn user that it would be mixed as not guaranteed when user pass 16.

is_fp16 = self.lightning_module.trainer.precision == 16

What would that check have to look like for a user custom plugin? what does is_fp16 refer to here and what do you need it to be in sharded? that the weights must be in half precision?

i think the intention is to check whether we are using ShardedNativeMixedPrecision, so is_mixed might be more precise name, and we could replace the check as precision == mixed for now.