Training Detail of MSD

[artony35]

Hi @tangy5,

Thank you for publishing the code of Swin UNETR and the pretrained weights! I am wondering what is the spacing used in task 07 pancreas to get 0.58 DSC for the tumor? It seems like the supplementary material only mentioned the data augmentation scheme and ROI sizes. Thank you so much!

[tangy5]

Hi @tangy5,

Thank you for publishing the code of Swin UNETR and the pretrained weights! I am wondering what is the spacing used in task 07 pancreas to get 0.58 DSC for the tumor? It seems like the supplementary material only mentioned the data augmentation scheme and ROI sizes. Thank you so much!

Hi @artony35 , thank you for the interest of the work! For the Task07, we used Spacing as isotropic 1mm for training, validation and testing. The validation result of Tumor Dice scores are around 0.51-0.58 according to different validation folds. Thank you !

[artony35]

Hi @tangy5,

Thank you for your clarification on training details!

We were able to run training and were having issues replicating the result. We used 646464 ROI and 111 spacing. It seems like the model was overfitting only on the pancreas task on both SSL and non SSL mode.
I wonder if you had overfitting issues specifically to Task07? The loss function looks like this:

Validation accuracy drops and never improved after the drop occurred. Overfitting always occur between 1000-2000 epoch.

We tried a few solution including adjusting the learning rate (2e-4/8e-4) as well as enabling droppath, but the loss graph looked similar.
We would love to have your input. Thank you for your time in advance!

[tangy5]

Hi @artony35 thanks for the follow-ups. Looking at the figure, I feel the training is impacted by corrupted gradients (The ditch), not the overfitting issue. It is typically caused by instable training (e.g., LR, optimizer, amp). Did you enable the amp?

[artony35]

Hi @tangy5 , thank you for your prompt reply! The AMP was on for the training. I attempted LR of 1 e-4, 2e-4 and 4e-4 and the ditch was still there. The optimizer was adamw as default. Do you think AMP might be corrupting the gradient?

[artony35]

Hi @tangy5, thank you for pointing me to the AMP. I have attempted 1 e-4, 2e-4 and 4e-4 with --noamp flag. It seems like there is still a ditch on all three attempts with the swin-unetr at around 1500-2000 epoch. I wonder if you guys used a different optimizer for task07? Maybe SGD? Thank you!

[artony35]

Here are the loss graphs of 1e-4, 2e-4 and 4e-4 LR. Interestingly when we tried unetr on task07, the loss goes to NaN at around the same epochs.