ViTAutoEnc nan

[chris-dem]

Does anyone have any clue why the ViTAutoEnc just gives me
RuntimeError: Function 'NativeDropoutBackward0' returned nan values in its 0th output..

1. Inputs are normalized and checked for nans or finite values
2. Gradient is clipped (Using lightning with max value 0.5)
3. Learning rate is very low (1e-5)
4. Loss function is non nan

[KumoLiu]

Hi @chris-dem, perhaps you can refer to these discussions.
https://discuss.pytorch.org/t/gradient-becomes-nan-with-random-runtimeerror-function-expbackward0-returned-nan-values-in-its-0th-output/166900
https://discuss.pytorch.org/t/logbackward-returned-nan-values-in-its-0th-output/92820
https://discuss.pytorch.org/t/training-with-mixed-precision-loss-is-nan-despite-finite-output-in-forward-pass/162937
Some hint the error may be due to a limited precision.

Hope it helps, thanks!