Deterministic=True works for version 1.1.4 , but encounters errors for version 1.5.5

[zhiqiangdon]

Hi,

I have found inconsistent behaviors about setting Deterministic=True in Trainer(). My code runs well when using Lightning with Version 1.1.4. However, after upgrading Lightning to Version 1.5.5, running the same code encounters the following error:

"RuntimeError: upsample_bilinear2d_backward_out_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation if that's acceptable for your application."

I am a little confused why it doesn't throw the error in Version 1.1.4. I understand that bilinear upsampling is not deterministic. Does Version 1.1.4 use deterministic algorithms only for the operations supporting that? Does Version 1.5.5 naively set all operations to use deterministic algorithms, not checking whether they support that or not? How can I use deterministic=True in Version 1.5.5 without worrying about errors?

Thanks,

[ananthsub]

In Lightning 1.5+, the framework uses torch.use_deterministic_algorithms which has more comprehensive determinism checks compared to the prior cudnn flag.

What errors are you seeing?

[zhiqiangdon]

Hi, the second paragraph in the above post is the error message. It seems that Lightning 1.5.5 doesn't check the availability for F.interpolate() function with the "bilinear" choice.