Running the same project in MONAI results in different outcomes

[alcarpo]

Hi,
I am running a project for automatic segmentation of stroke lesion in the brain using SegResNet and the ATLAS v2.0 dataset.
However, when I run the same notebook multiple times, I always get different Mean Dice results at the end. I have ensured "set_determinism(seed=0)" and "random.seed(6)" for reproducible purposes and for instance, when applying random hold-out partition into training and validation subsets, with this I always get the same indices (images) for each subset. That is why I am not sure why then the results of a typical PyTorch training process are different when running various times.

Could this be due to the Randmon transforms applied at the training dataset? If so, wouldn't be this covered by setting "set_determinism(seed=0)" and "random.seed(6)"?

I look forward for your comments.
Automatic_Segmentation_Stroke_Lesions_by SPECs Research Group.zip

Thanks :)

[wyli]

possibly because of torch.backends.cudnn.benchmark = True, please see http://man.hubwiz.com/docset/PyTorch.docset/Contents/Resources/Documents/notes/randomness.html#cudnn

[alcarpo]

Then torch.backends.cudnn.benchmark should be set up as FALSE??

[wyli]

yes, actually the command monai.utils.set_determinism(0) will set the flags for you. The additional torch.backends.cudnn.benchmark = True in your script might cause the non-deterministic behaviour.

[alcarpo]

Okay, thank you! I will execute it several times to see if the results are mantained and I will get back here to inform the result. If solved, it might help others too

[alcarpo]

Hi, I have tried to do it and still get different results when I run the model several times

[wyli]

ok, otherwise maybe it's related to the upsampling of segresnet

MONAI/monai/networks/nets/segresnet.py

Lines 47 to 49 in f08f1d3

	upsample_mode: [``"deconv"``, ``"nontrainable"``, ``"pixelshuffle"``]
	The mode of upsampling manipulations.
	Using the ``nontrainable`` modes cannot guarantee the model's reproducibility. Defaults to``nontrainable``.

perhaps you can use a different option or network just for a sanity check.