Confusion with metrics and folder structure

[Mirko1998]

Hello Anomalib team,
first of all, thank you for your very practical library and the work you have put into it.
Regarding the folder structure:
So for training only normal_dir is used? (from the config)
For testing and evaluating then:
abnormal_dir: , normal_test_dir: , mask:?
Unfortunately, I couldn't quite read this out of the documentation. So the train images are all normal images? and the test images have masks for the metrc evaluation?
The metrics for (image and pixel) AUROC and F1 are both from Pytorch and I can read their exact function there or are they partially adapted here?
Thank you very much for your feedback :)

[samet-akcay]

Hi @Mirko1998, thank you for your interest in the library.

Folder Dataset

Folder dataset is highly customisable, so it might be a bit confusing.

normal_test_dir is an optional argument. If it is set, normal_dir will only be used for training. Otherwise, normal_dir will be split into train/val/test sets for training and performance evaluation.

abnormal_dir is used for testing and evaluation.

mask_dir is also optional and depends on the task type. If the task is segmentation and you have mask_dir, Folder dataset will ensure that these are properly set. If mask_dir is None, then this means the task is a classification task.

We have a Jupyter notebook for the Folder dataset. Can you have a look at it and tell us if there is anything missing or unclear there?

Metrics

As I mentioned above, if you have a mask directory, it is possible to report segmentation metrics. Otherwise, only classification metrics will be reported.

We use metrics from the torchmetrics library, and also have some customised metrics within anomalib itself. For example, you could refer to this link to see the metrics in anomalib
https://github.com/openvinotoolkit/anomalib/tree/main/src/anomalib/utils/metrics

In theory, if you add a metric from torch metric library to your config file, anomalib should run it. If you see a metric that does not work, you could report here.

Documentation

We are aware that some of these bits might be missing in our documentation. This is mainly because we are currently developing v1.0, which we plan to release some time in October. This slowed down our documentation efforts for the existing versions. Things will hopefully be a lot more clear in v1.0 documentation. Thanks again!

[pirnerjonas]

Hi @samet-akcay,

In theory, if you add a metric from torch metric library to your config file, anomalib should run it. If you see a metric that does not work, you could report here.

could you elaborate on how to specifically add a metric like ConfusionMatrix which is not part of anomalibs metrics within a config.yml?