After training with augmentation, inference of same image multiple times gives different result

[TurboJonte]

I've been doing some experiments with training a PaDiM model on the hazelnut MVTech dataset, until now I have trained without using any augmentations. I now wanted to learn and try how to use an augmented dataset. I've tried to add the same augmentations as in the original paper.
When trying out the model trained with an augmented dataset I now get different result when doing inference multiple times on the same image. And it's the actual transforms being used to augment the images that is now visible in the resulted heatmap overlayed on the image. How come?

PaDiM_inference.mp4

This is my training script:

# Import the required modules
#from anomalib.data import MVTec
import torchvision.transforms.v2 as transforms
from anomalib.data import Folder
from anomalib.models import Padim
from anomalib.engine import Engine
from anomalib import TaskType
from pathlib import Path

# Initialize the datamodule, model and engine

def main():
    transform = transforms.Compose([
                                    transforms.RandomRotation(degrees=(-2, 2)),
                                    transforms.Resize(size=(292,292)),
                                    transforms.RandomCrop(size=(282,282)),
                                    transforms.CenterCrop(size=(256,256))
                                    ])
    datamodule = Folder(name='hazelnut_1.0.0.3', normal_dir=Path('hazelnut\\train\\good'), task=TaskType.CLASSIFICATION, train_transform=transform, train_batch_size=8)
    model = Padim(layers=["layer1", "layer2", "layer3"])
    engine = Engine(devices='1')

    # Train the model
    engine.fit(datamodule=datamodule, model=model)

if __name__ == "__main__":
    main()

The model has been converted to PyTorch with the following script:

from anomalib.deploy import ExportType, OpenVINOInferencer
from anomalib.engine import Engine
from anomalib.models import Padim
from anomalib import TaskType

model = Padim()
engine = Engine(task=TaskType.SEGMENTATION, devices='1')
engine.export(model=model, export_type=ExportType.TORCH, ckpt_path="results\\Padim\\hazelnut_1.0.0.3\\v1\\weights\\lightning\\model.ckpt", export_root='C:\\Temp\\hazelnut_1.0.0.3_torch')

This is the script used for inference:

from anomalib.deploy import ExportType, OpenVINOInferencer, TorchInferencer
from anomalib.engine import Engine
from anomalib.models import Padim
from anomalib import TaskType
from torchvision.transforms.v2.functional import to_image

self.model = TorchInferencer(path=pytorch_model_path,  # Path to the OpenVINO model.
                                    device=device # CPU or CUDA
                                    )
pred = self.model.predict(image=img_tensor)

[TurboJonte]

@samet-akcay Do you have any comment on this?

[samet-akcay]

@djdameln FYI

[djdameln]

Hi, thanks for pointing this out.

The short answer is: You need to pass an eval_transform to the datamodule before training, which defines how the images should be transformed during inference. In line with the training transforms in your example, the following would be sufficient:

transforms.Compose([
    transforms.Resize(size=(292,292)),
    transforms.CenterCrop(size=(256,256))
])

Note that it would be better to also normalize the input images (both train and eval) to ImageNet statistics, because Padim's pre-trained backbone was trained on ImageNet:

transform.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

---

When only the train_transform is specified, Anomalib will re-use that transform for evaluation/inference to ensure that there is no difference in the scaling and normalization characteristics of the images between training and inference. In your example, Anomalib re-used the training transform during inference, including the random rotations and random cropping, resulting in the non-determinstic behaviour you observed.

The motivation for re-using the train transforms during inference is that there is no other way of ensuring that the inference runs with the same input shape and normalization statistics as the training. The alternatives, not applying any transforms, or applying the default model-specific transforms, would both break inference due to input shape mismatch.

These modules were changed very recently and we are still working on the documentation for this. Reading your post, I realize that the transform behaviour could be confusing to users, so we'll discuss internally if any changes are needed.