Parchcore Data Augmentation Example

[nixczhou]

Hi, may I ask the example of how to add data augmentation to add more data to train patchcore? Cause it is the memory bank base algo, so I guess it is better to has lots of data for it. I could not find any documentation to create that yaml file. What I want to do:

1. Auto Brightness and Contrast
2. Rotation
3. Random Crop

[djdameln]

Hi, custom augmentations can be added to a training pipeline by providing the path to an albumentations config file in the dataset.transform_config section of the config.yaml. A different set of transforms/augmentations may be provided for the training and evaluation dataloaders. This page of the albumentations documentation explains how to create a .yaml file.

Please note that there are a few mandatory transforms that you will always have to add to your custom transforms file. First of all, the Resize and ToTensorV2 transforms are always required, because these make sure that the images are passed to the model in the right format. In addition, for most models it is needed to normalize to the ImageNet statistics using the Normalize transform. When no custom transforms file is provided, Anomalib adds these default transforms automatically.

So, for your use case, you could achieve your desired augmentations by setting dataset.transform_config.train to the path to a .yaml file with the following contents:

__version__: 1.3.0
transform:
  __class_fullname__: Compose
  additional_targets: {}
  bbox_params: null
  keypoint_params: null
  p: 1.0
  transforms:
  - __class_fullname__: Resize
    always_apply: true
    height: 256
    interpolation: 1
    p: 1
    width: 256
  - __class_fullname__: RandomBrightnessContrast
    always_apply: false
    brightness_by_max: true
    brightness_limit:
    - -0.2
    - 0.2
    contrast_limit:
    - -0.2
    - 0.2
    p: 0.5
  - __class_fullname__: Rotate
    always_apply: false
    border_mode: 4
    crop_border: false
    interpolation: 1
    limit:
    - -10
    - 10
    mask_value: null
    p: 0.5
    rotate_method: largest_box
    value: null
  - __class_fullname__: RandomCrop
    always_apply: true
    height: 224
    p: 1.0
    width: 224
  - __class_fullname__: Normalize
    always_apply: true
    max_pixel_value: 255.0
    mean:
    - 0.485
    - 0.456
    - 0.406
    p: 1.0
    std:
    - 0.229
    - 0.224
    - 0.225
  - __class_fullname__: ToTensorV2
    always_apply: true
    p: 1.0
    transpose_mask: false

which was generated by running the following code:

import albumentations as A
from albumentations.pytorch import ToTensorV2

transform = A.Compose(
    [
        A.Resize(256, 256, always_apply=True),
        A.RandomBrightnessContrast(),
        A.Rotate((-10,10)),
        A.RandomCrop(224, 224, always_apply=True),
        A.Normalize(mean=(0.485, 0.456, 0.406),
        ToTensorV2()
    ]
)

A.save(transform, "patchcore_augmentations.yaml", data_format="yaml")

[monkeycc]

transform = A.Compose(
    [
        A.Resize(256, 256, always_apply=True),
        A.RandomBrightnessContrast(),
        A.Rotate((-10,10)),
        A.RandomCrop(224, 224, always_apply=True),
        A.Normalize(mean=(0.485, 0.456, 0.406)),
        ToTensorV2()
    ]
)

Thank you for your code
But I found a small mistake

[djdameln]

Thanks, I've updated my answer

[monkeycc]

config.yaml

  transform_config:
    train: D:/anomalib/anomalib/models/cflow/patchcore_augmentations.yaml
    val: D:/anomalib/anomalib/models/cflow/patchcore_augmentations.yaml

I don't know whether to call it like this ？
@djdameln

[samet-akcay]

@monkeycc, yes, it should be something like that. For a reference, you could refer to the DRAEM configs here
https://github.com/openvinotoolkit/anomalib/blob/764f97dd3bb6f082a51422a946a2582edf07d511/anomalib/models/draem/config.yaml#L11-L13

[monkeycc]

  format: mvtec
  path: ./mvtec_anomaly_detection
  category: bottle
  task: segmentation
  transform_config:
    train: D:/anomalib/anomalib/models/cflow/patchcore_augmentations.yaml
    val: D:/anomalib/anomalib/models/cflow/patchcore_augmentations.yaml

import albumentations as A
from albumentations.pytorch import ToTensorV2

transform = A.Compose(
    [
        A.VerticalFlip (),
        A.HorizontalFlip (),
        A.Transpose(),
        A.Rotate (limit=45, interpolation=1, border_mode=1, value=None, mask_value=None, rotate_method='largest_box', crop_border=False, always_apply=False, p=0.5),
        A.HueSaturationValue (hue_shift_limit=20, sat_shift_limit=30, val_shift_limit=20, always_apply=False, p=0.5),
        A.RandomBrightnessContrast (brightness_limit=0.2, contrast_limit=0.2, brightness_by_max=True, always_apply=False, p=0.5),
        ToTensorV2()
    ]
)

A.save(transform, "patchcore_augmentations.yaml", data_format="yaml")

__version__: 1.3.0
transform:
  __class_fullname__: Compose
  additional_targets: {}
  bbox_params: null
  keypoint_params: null
  p: 1.0
  transforms:
  - __class_fullname__: VerticalFlip
    always_apply: false
    p: 0.5
  - __class_fullname__: HorizontalFlip
    always_apply: false
    p: 0.5
  - __class_fullname__: Transpose
    always_apply: false
    p: 0.5
  - __class_fullname__: Rotate
    always_apply: false
    border_mode: 1
    crop_border: false
    interpolation: 1
    limit:
    - -45
    - 45
    mask_value: null
    p: 0.5
    rotate_method: largest_box
    value: null
  - __class_fullname__: HueSaturationValue
    always_apply: false
    hue_shift_limit:
    - -20
    - 20
    p: 0.5
    sat_shift_limit:
    - -30
    - 30
    val_shift_limit:
    - -20
    - 20
  - __class_fullname__: RandomBrightnessContrast
    always_apply: false
    brightness_by_max: true
    brightness_limit:
    - -0.2
    - 0.2
    contrast_limit:
    - -0.2
    - 0.2
    p: 0.5
  - __class_fullname__: ToTensorV2
    always_apply: true
    p: 1.0
    transpose_mask: false

python tools/train.py --model cflow

  | Name                  | Type                     | Params
-------------------------------------------------------------------
0 | image_threshold       | AnomalyScoreThreshold    | 0
1 | pixel_threshold       | AnomalyScoreThreshold    | 0
2 | model                 | CflowModel               | 236 M
3 | image_metrics         | AnomalibMetricCollection | 0
4 | pixel_metrics         | AnomalibMetricCollection | 0
5 | normalization_metrics | MinMax                   | 0
-------------------------------------------------------------------
81.6 M    Trainable params
154 M     Non-trainable params
236 M     Total params
946.249   Total estimated model params size (MB)
Epoch 0:   0%|                                                                                  | 0/10 [00:00<?, ?it/s]Traceback (most recent call last):
  File "tools/train.py", line 71, in <module>
    train()
  File "tools/train.py", line 60, in train
    trainer.fit(model=model, datamodule=datamodule)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 770, in fit
    self._call_and_handle_interrupt(
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 723, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 811, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1236, in _run
    results = self._run_stage()
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1323, in _run_stage
    return self._run_train()
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1353, in _run_train
    self.fit_loop.run()
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 266, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 208, in advance
    batch_output = self.batch_loop.run(batch, batch_idx)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\batch\training_batch_loop.py", line 90, in advance
    outputs = self.manual_loop.run(split_batch, batch_idx)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\optimization\manual_loop.py", line 115, in advance
    training_step_output = self.trainer._call_strategy_hook("training_step", *step_kwargs.values())
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1765, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 333, in training_step
    return self.model.training_step(*args, **kwargs)
  File "D:\0SDXX\anomalib\anomalib\models\cflow\lightning_model.py", line 104, in training_step
    activation = self.model.encoder(images)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\0SDXX\anomalib\anomalib\models\components\feature_extractors\timm.py", line 101, in forward
    features = dict(zip(self.layers, self.feature_extractor(inputs)))
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\timm\models\features.py", line 232, in forward
    return list(self._collect(x).values())
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\timm\models\features.py", line 203, in _collect
    x = module(x)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\torch\nn\modules\conv.py", line 457, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "D:\anaconda\envs\anomalib_env\lib\site-packages\torch\nn\modules\conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same
Epoch 0:   0%|          | 0/10 [00:25<?, ?it/s]

@samet-akcay

[lekshmijk]

I'm getting the same error @monkeycc were you able to fix the issue?

[monkeycc]

transform = A.Compose(
    [
        A.Resize(256, 256, always_apply=True),
        A.RandomBrightnessContrast(),
        A.Rotate((-10,10)),
        A.RandomCrop(224, 224, always_apply=True),
        A.Normalize(mean=(0.485, 0.456, 0.406)),
        ToTensorV2()
    ]
)

These will not report errors

But if you change the color, change the direction
Error will be reported

[ngadhvi]

@samet-akcay Does this generate a separate augmented data set on the fly and add it to the original dataset or it replaces the original data set with the augmentations mentioned in the .yaml and if you could please explain by what factor?