Order of samples being evaluated changes reported dice score

[bip5]

Hi guys,
I've come across a weird issue recently, that is changing the the order in which the samples are evaluated seems to change their dice scores? I've tried it multiple times and the same order leads to the same results. My evaluation pipeline has model.eval() and inference happens inside torch.no_grad() so I don't understand how this is happenning. Anyone come across anything similar before?

[KumoLiu]

Hi @bip5, thanks for reporting. Could you please help share a small piece of code that could help reproduce the issue then we may help take a deep look into it? Thanks!

[bip5]

Hi YunLiu, Thank you for the reply, I'd have to reply with the whole pipeline for this, but I'm not sure it that's appropriate since you're probably after some simpler way to replicate. If this is a genuine issue, one simple way to replicate this would be to turn shuffle off during evaluation. Following this where you'd normally list the file paths for the dataset class you could use sorted for the first attempt and then sorted(paths, reverse=True) for the second evaluation attempt. If this doesn't replicate the issue please let me know and I'll try to find a simple representation of my problem. I am using monai 0.9.0, so that could potentially have something to do with it.. Please let me know your thoughts. Thank you

…

________________________________ From: YunLiu ***@***.***> Sent: Monday, November 6, 2023 8:31 AM To: Project-MONAI/MONAI ***@***.***> Cc: Bishnu Paudel [bip5] ***@***.***>; Mention ***@***.***> Subject: Re: [Project-MONAI/MONAI] Order of samples being evaluated changes reported dice score (Discussion #7202) [RHYBUDD! E-BOST ALLANOL / CAUTION! EXTERNAL E-MAIL] Hi @bip5<https://github.com/bip5>, thanks for reporting. Could you please help share a small piece of code that could help reproduce the issue then we may help take a deep look into it? Thanks! — Reply to this email directly, view it on GitHub<#7202 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARJRZPM5UZGIF7UMUAQQVO3YDCN57AVCNFSM6AAAAAA66UTI4WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TIOBUG4ZTS>. You are receiving this because you were mentioned.Message ID: ***@***.***>

---------------------------------------------------------------------------------------------------------------------- Y Brifysgol orau yn y DU am Ansawdd ei Dysgu a Phrofiad Myfyrwyr Best University in the UK for Teaching Quality and Student Experience (The Times and Sunday Times, Good University Guide 2021)

[KumoLiu]

Hi @bip5, It would be great if you could try it out with the latest version of monai, and I'll find some time to try it out soon to give you feedback, thanks!

[bip5]

Hi YunLiu, For some reason my pretrained model from 0.9.0 doesn't work at all with the latest version. I added ToMetaTensorD to my transforms but that didn't seem to help. Regardless, I trained a new model with 1.3.0 (with the data_array` is not of type MetaTensor, assuming affine to be identity.​ warning present..) and the issue is still there Thank you

…

________________________________ From: YunLiu ***@***.***> Sent: Tuesday, November 7, 2023 2:52 AM To: Project-MONAI/MONAI ***@***.***> Cc: Bishnu Paudel [bip5] ***@***.***>; Mention ***@***.***> Subject: Re: [Project-MONAI/MONAI] Order of samples being evaluated changes reported dice score (Discussion #7202) [RHYBUDD! E-BOST ALLANOL / CAUTION! EXTERNAL E-MAIL] Hi @bip5<https://github.com/bip5>, It would be great if you could try it out with the latest version of monai, and I'll find some time to try it out soon to give you feedback, thanks! — Reply to this email directly, view it on GitHub<#7202 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARJRZPIOCGKSNUR63PWFXKDYDGO6NAVCNFSM6AAAAAA66UTI4WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TIOJUGEYDC>. You are receiving this because you were mentioned.Message ID: ***@***.***>

---------------------------------------------------------------------------------------------------------------------- Y Brifysgol orau yn y DU am Ansawdd ei Dysgu a Phrofiad Myfyrwyr Best University in the UK for Teaching Quality and Student Experience (The Times and Sunday Times, Good University Guide 2021)

[bip5]

HI YunLiu, I fixed the errors related to MetaTensor and my previous models are now evaluated okay. The performance issue relating to evaluation order is still present. Please could you confirm if this is a genuine issue? To reverse the order of evaluation I used the Subset class from torch.utils.data and reversed the order of indices test_ds = Subset(test_ds,test_indices[::-1]) Thank you Kind regards, Bishnu

…

________________________________ From: Bishnu Paudel [bip5] ***@***.***> Sent: Tuesday, November 7, 2023 11:20 AM To: Project-MONAI/MONAI ***@***.***>; Project-MONAI/MONAI ***@***.***> Cc: Mention ***@***.***> Subject: Re: [Project-MONAI/MONAI] Order of samples being evaluated changes reported dice score (Discussion #7202) Hi YunLiu, For some reason my pretrained model from 0.9.0 doesn't work at all with the latest version. I added ToMetaTensorD to my transforms but that didn't seem to help. Regardless, I trained a new model with 1.3.0 (with the data_array` is not of type MetaTensor, assuming affine to be identity.​ warning present..) and the issue is still there Thank you

________________________________ From: YunLiu ***@***.***> Sent: Tuesday, November 7, 2023 2:52 AM To: Project-MONAI/MONAI ***@***.***> Cc: Bishnu Paudel [bip5] ***@***.***>; Mention ***@***.***> Subject: Re: [Project-MONAI/MONAI] Order of samples being evaluated changes reported dice score (Discussion #7202) [RHYBUDD! E-BOST ALLANOL / CAUTION! EXTERNAL E-MAIL] Hi @bip5<https://github.com/bip5>, It would be great if you could try it out with the latest version of monai, and I'll find some time to try it out soon to give you feedback, thanks! — Reply to this email directly, view it on GitHub<#7202 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARJRZPIOCGKSNUR63PWFXKDYDGO6NAVCNFSM6AAAAAA66UTI4WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TIOJUGEYDC>. You are receiving this because you were mentioned.Message ID: ***@***.***>

---------------------------------------------------------------------------------------------------------------------- Y Brifysgol orau yn y DU am Ansawdd ei Dysgu a Phrofiad Myfyrwyr Best University in the UK for Teaching Quality and Student Experience (The Times and Sunday Times, Good University Guide 2021)

[bip5]

Hi guys, Just wanted to add I was able to replicate this issue using SwinUNETR as well. Please let me know whether or not you see the same behaviour with your dataset. sample results(5/150 samples) Subject ID Original index Dice score 00011-000 619 0.9273 00014-000 567 0.8223 00020-000 364 0.92 00045-001 1189 0.9352 00051-000 858 0.8665 When the order of processing for evaluation is reversed the performance on same samples is much lower: Subject ID Original index Dice score 00011-000 619 0.7626 00014-000 567 0.7589 00020-000 364 0.9042 00045-001 1189 0.9014 00051-000 858 0.7599 Kind regards, Bishnu

…

________________________________ From: Bishnu Paudel [bip5] ***@***.***> Sent: Wednesday, November 8, 2023 11:12 PM To: Project-MONAI/MONAI ***@***.***>; Project-MONAI/MONAI ***@***.***> Cc: Mention ***@***.***> Subject: Re: [Project-MONAI/MONAI] Order of samples being evaluated changes reported dice score (Discussion #7202) HI YunLiu, I fixed the errors related to MetaTensor and my previous models are now evaluated okay. The performance issue relating to evaluation order is still present. Please could you confirm if this is a genuine issue? To reverse the order of evaluation I used the Subset class from torch.utils.data and reversed the order of indices test_ds = Subset(test_ds,test_indices[::-1]) Thank you Kind regards, Bishnu

________________________________ From: Bishnu Paudel [bip5] ***@***.***> Sent: Tuesday, November 7, 2023 11:20 AM To: Project-MONAI/MONAI ***@***.***>; Project-MONAI/MONAI ***@***.***> Cc: Mention ***@***.***> Subject: Re: [Project-MONAI/MONAI] Order of samples being evaluated changes reported dice score (Discussion #7202) Hi YunLiu, For some reason my pretrained model from 0.9.0 doesn't work at all with the latest version. I added ToMetaTensorD to my transforms but that didn't seem to help. Regardless, I trained a new model with 1.3.0 (with the data_array` is not of type MetaTensor, assuming affine to be identity.​ warning present..) and the issue is still there Thank you

________________________________ From: YunLiu ***@***.***> Sent: Tuesday, November 7, 2023 2:52 AM To: Project-MONAI/MONAI ***@***.***> Cc: Bishnu Paudel [bip5] ***@***.***>; Mention ***@***.***> Subject: Re: [Project-MONAI/MONAI] Order of samples being evaluated changes reported dice score (Discussion #7202) [RHYBUDD! E-BOST ALLANOL / CAUTION! EXTERNAL E-MAIL] Hi @bip5<https://github.com/bip5>, It would be great if you could try it out with the latest version of monai, and I'll find some time to try it out soon to give you feedback, thanks! — Reply to this email directly, view it on GitHub<#7202 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARJRZPIOCGKSNUR63PWFXKDYDGO6NAVCNFSM6AAAAAA66UTI4WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TIOJUGEYDC>. You are receiving this because you were mentioned.Message ID: ***@***.***>

---------------------------------------------------------------------------------------------------------------------- Y Brifysgol orau yn y DU am Ansawdd ei Dysgu a Phrofiad Myfyrwyr Best University in the UK for Teaching Quality and Student Experience (The Times and Sunday Times, Good University Guide 2021)

[KumoLiu]

Hi @bip5, I finally found the time to test it, and I used the spleen bundle from the Model Zoo repo. I didn't find any issues.
A possible reason may be there are some non-deterministic PyTorch operations.
Please refer to https://pytorch.org/docs/stable/notes/randomness.html#reproducibility for more details about reproducibility.
Thanks!

[bip5]

Hi @KumoLiu ,

Thank you for your confirmation. This helped me locate some errors in my pipeline relating to my use of DiceMetric to report individual scores and also with the indexing to attribute those scores to the appropriate samples. After the issues are sorted the scores are consistent(at least up to 5 dec points)