How to flag certain modules as non-deterministic

[dsethz]

Hey,

Question: How can I set a module/layer in my model-class to always be non-deterministic (irrespective of the deterministic flag in pl.Trainer())?

Context: I use pl to train a simple AutoEncoder that uses bilinear upscaling in the decoder part. For debugging, I use the deterministic flag of the pl.Trainer(). However, I receive the following error

RuntimeError: upsample_bilinear2d_backward_out_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.

Unfortunately, the error does not hint at how to set the module to be non-deterministic and neither does the documentation.

Cheers
dsethz

[dsethz]

Any feedback on this issue?

[akihironitta]

Hi @dsethz!

The error comes from PyTorch, but not from Lightning, and I think it's not (shouldn't be) feasible even in pure PyTorch because the flag is for reproducibility and if you allow randomness in certain layers, you can't reproduce the same result anymore.

https://pytorch.org/docs/stable/notes/randomness.html

[dsethz]

Thank you for your feedback.

I see, that makes sense.

[akihironitta]

Hi @dsethz! Just an update that you might be interested in :)

PyTorch 1.11 (the latest release) supports raising a warning instead of RuntimeError in case any of operations don't have a deterministic implementation, which is documented in https://pytorch.org/docs/1.11/generated/torch.use_deterministic_algorithms.html.

torch.use_deterministic_algorithms(True, warn_only=True)

There's a possibility of integrating it in Lightning in the future. Here's an open PR: #12588

---

The feature should land in master, and will be released in the 1.7.0 release which is a few months away from now :)

[dsethz]

That's cool, thank you for the update!