Difference in UniNet MVTec I-AUROC between anomalib benchmarking and official paper

[GunelLuo]

First of all, thank you for the great work on anomalib and for including UniNet in the benchmarking.

I noticed that all scores for UniNet on the MVTec AD dataset in the anomalib benchmarking results are lower than those reported in the official UniNet paper. Could you please share the differences in training/inference setup that might explain the gap.

Thanks in advance!

[rajeshgangireddy]

Hi @GunelLuo
@ashwinvaidya17 Implemented the UniNet model. He has some observations and can answer your question after he is back from his break in a week or so.

[ashwinvaidya17]

@GunelLuo
The implementation in Anomalib is based on the codebase here https://github.com/pangdatangtt/UniNet
There are a couple of reasons for the discrepancy.

1. In Anomalib, we collect the benchmark for all the models based on the same seed. In papers, numbers are generally collected from average of multiple runs or best among multiple runs. This might explain the difference in numbers.
2. The reference codebase uses dataset and category specific parameters in some cases. For instance, https://github.com/pangdatangtt/UniNet/blob/0c211bacca7070b4b46a6841bb03b47800e25fd2/main.py#L122
   https://github.com/pangdatangtt/UniNet/blob/0c211bacca7070b4b46a6841bb03b47800e25fd2/train_unsupervisedAD.py#L44
3. According to other users, the reference code still does not match the paper results out of the box Reproduce results in MVTec AD pangdatangtt/UniNet#16