training on gpu becomes non-deterministic

[2533245542]

I usually set seed_everything() at the beginning of my script but this does not always solves the problem.

When I train models on cpu, it is determinstic; but when I switch to gpu, it becomes non-deterministic.

When I am training simple model, like one-layer LSTM, it is deterministic both on cpu and gpu.

But when I train a more completed model like LSTM-FCN, it is deterministic on cpu but not on gpu

Can I get any help on debugging?

I got my LSTM-FCN model from here (https://github.com/timeseriesAI/tsai/blob/main/tsai/models/RNN_FCN.py) and the LSTM model I tested was simply a nn.LSTM

[rohitgr7]

maybe setting Trainer(deterministic=True) might help?

[2533245542]

Wow, it worked. This is amazing. I thought seed_everything() has done everything pytorch lightning could do. Is there any documentation for the Trainer(deterministic=True) you mentioned? Want to take a look.

[rohitgr7]

sure! here are the docs: https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#deterministic

in short deterministic operations are slower so it's not enabled by default.

[yipliu]

Hi,

Even if they are both on the GPU, seed_everything does not give me near the same results for my model.

Adding the tag Trainer(deterministic=True) currently makes all my results repeatable