Correct way to train image without anomaly image

[ntdgo]

Hi, I am interested in ganomaly model to detect anomaly images based on training only normal images.

I have referred to this topic #192 to set threshold to manual. But I am not sure this is the correct way or not.

This is my config data and file structure:
My dataset is named as wallcrack

anomalib/
├─ datasets/
│  ├─ wallcrack/
│  │  ├─ good/
│  │  │  ├─ image_0.jpg
│  │  │  ├─ ...
│  │  │  ├─ image_n.jpg
│  │  ├─ bad/
│  │  │  ├─ good_image_also.jpg
│  │  ├─ config.yaml/

config.yaml

dataset:
  name: folder #options: [mvtec, btech, folder]
  format: folder
  path: ./datasets/wallcrack
  normal_dir: good
  abnormal_dir: bad 
  ...

metrics:
  ...
  threshold:
    method: manual #manual: [adaptive, manual]
    manual_image: 0.5
    manual_pixel: 0.5

Command: anomalib>python tools/train.py --model ganomaly --config datasets/wallcrack/config.yaml

Could anyone please confirm this is the correct way?

Thanks, and best regards!

[djdameln]

In the current version of Anomalib (v0.3.7), the Folder dataset format always requires that a folder of abnormal/anomalous images is provided in dataset.abnormal_dir. These images are used during validation to compute the optimal threshold value, and during testing to rate the performance of the trained model.

By placing normal images to the abnormal_dir folder, you circumvent the checks that prevent Anomalib from running without abnormal images. This workaround is effective, but please be aware that the optimal threshold value, as well as any evaluation metrics computed during testing, would be meaningless, because incorrect ground truth labels are assigned to the images found in bad/. Since you're setting a manual threshold, you don't need to worry about the value of the adaptive threshold in this case. To prevent Anomalib from computing wrong and possibly misleading performance numbers as a result of the incorrect labels, I would suggest to remove all image- and pixel-level metrics from your configuration. So the metrics section of your config would look like this:

metrics:
  image: []
  pixel: []
  threshold:
    method: manual #options: [adaptive, manual]
    manual_image: 0.5
    manual_pixel: 0.5

---

The Folder dataset was originally designed with the assumption that each dataset would always contain at least a few abnormal/anomalous images for validation and evaluation purposes. We realize that this may be too strict, and some users might be interested in just training a model on normal images, without running any validation or evaluation afterwards. We are currently working on a redesign of the data classes that would allow this. This new design will most likely be included in the upcoming release (If you're interested in the changes, you could check out this branch). Be aware that there will be some changes to the dataset section of the config, so you might be required to update your config files to the new format.

[ntdgo]

Hi @djdameln, as the metric set to null, I have problem with early_stopping config.

If I keep the early_stopping as:

model:
  early_stopping:
    patience: 3
    metric: image_AUROC
    mode: max

→ RuntimeError: Early stopping conditioned on metric image_AUROC which is not available. Pass in or modify your EarlyStopping callback to use any of the following: ``

If I remove early_stopping as:

model:
  early_stopping:
    patience: 20
    metric: []
    mode: max

→ Error: RuntimeError: Early stopping conditioned on metric [] which is not available. Pass in or modify your EarlyStopping callback to use any of the following: ``

How can I get over this problem?
Thank you!

[djdameln]

In Anomalib, early stopping is used by models that involve fine-tuning of the neural network weights. To determine when the training has converged, we monitor some performance metric computed over the validation set. We stop the training once the value of the metric no longer increases for a certain number of epochs. This number is referred to as the 'patience' of the early stopping mechanism.

With your setup, it would not be a good idea to condition early stopping on a validation metric, because as explained earlier, the value of the performance metrics would be meaningless due to wrong labels in the ground truth annotations. In this case you would have two options:

1. You could condition the early stopping on the training loss, by setting the early_stopping.metric parameter to train_loss_epoch. Please be aware that it is generally not advisible to condition early stopping on trainings loss, as it could lead to overfitting. But due to a lack of other meaningful metrics, it could be an option in your setup. Make sure you set early_stopping.mode to min, to indicate that the we are looking to minimize, not maximize the loss.

2. You could run without early stopping. While early stopping cannot be disabled explicitly (this is another WIP), you could set the early stopping patience to a value higher than the maximum number of training epochs (which can be configured in trainer.max_epochs). This way, the early stopping condition would never be reached, and the training will always run for the fixed maximum number of epochs.

For example, you could use the following config to ensure that your model trains for at most 20 epochs, and training will be stopped early when the training loss does not decrease for 3 consecutive epochs. To disable early stopping altogether, just set patience to a value of 20 or higher.

model:
  ...
  early_stopping:
    patience: 3
    metric: train_loss_epoch
    mode: min
  ...

trainer:
  ...
  max_epochs: 20
  ...

Note: you might need to pull the latest changes, because we recently merged a PR to properly log the training loss.

Hope that's clear. Please do not hesitate to reach out if you have any additional questions.

[ntdgo]

Currently, I am using your branch: https://github.com/openvinotoolkit/anomalib/tree/feature/datamodules. I will add some log command as the implementation of #804 to monitor the loss.

In addition, is there any option to continue the job training? For example, in the first training, I set max_epochs: 20 to train 20 epochs, then I want to train 10 epochs more.

→ solved by replace:
https://github.com/openvinotoolkit/anomalib/blob/8a4e46c4c43cb350b26c4c86b700fb463a6c3809/anomalib/utils/callbacks/__init__.py#L62-L68
and
https://github.com/openvinotoolkit/anomalib/blob/8a4e46c4c43cb350b26c4c86b700fb463a6c3809/anomalib/utils/cli/cli.py#L192-L198
by

checkpoint = ModelCheckpoint(
        dirpath=os.path.join(config.project.path, "weights"),
        filename='{epoch}-{val_loss:.2f}',
        monitor=monitor_metric,
        mode=monitor_mode,
        every_n_epochs=1,
        save_top_k=-1,
        auto_insert_metric_name=False,
    )

Thank you!

[SebastienThibert]

Hi,

is there any update regarding training on a custom dataset with good/normal images only ?

[djdameln]

Yes, it is now possible to train on a dataset with only normal images. When configuring the folder dataset, just specify only the path to the normal images. Please note that adaptive threshold computation will not work as intended in this case, because the F1 score cannot be computed when the validation set does not contain anomalous images. Anomalib will also not be able to report any meaningful evaluation metrics.

Here's an example how you could configure the directory structure in the dataset config:

dataset:
  name: custom_dataset
  format: folder
  root: path/to/your/dataset
  normal_dir: normal_images
  abnormal_dir: null
  normal_test_dir: null
  mask_dir: null

[arkerman]

Hi, @djdameln . Sorry to bother the communication. I have a question about the pixel_threshold. If i set the threshold "adaptive", how does pixel_threshold be calculated? I checked the source code. But i don't obtain a clear idea. Is pixel_threshold related to the maximum f1_score? Any suggestion will be appreciated!

[djdameln]

Hi, the class that computes the adaptive threshold is the AnomalyScoreThreshold, which can be found here. The adaptive threshold is computed by maximizing the F1 score over the validation set. It works by collecting the model predictions and the ground truth labels, and then computes the precision and recall for a range of threshold values. The precision and recall values are then used to determine the F1 scores, after which the threshold is selected that corresponds to the maximum observed F1 score.

For the image threshold, the ground truth labels are the binary image labels (normal vs anomalous) and the predictions are the anomaly scores produced by the model. For the pixel threshold, the ground truth labels are the binary anomaly masks and the predictions are the pixel-level anomaly maps produced by the model.

[arkerman]

Thanks a lot, buddy! Really appreciate your reply! I'll debug the source code again. Thanks again for your explanation!