GPU out of memory using PatchCore e PaDim

[LorenzoF6]

Hi,
i'm trying to test Patchcore e Padim with my own dataset (1000 good images / 1400 defect images) and using different combination of batch_size and num_workers, even small numbers such as 4 or 2, my models go every time in out of memory.
How can I resolve the issue?
Thansk

[rajeshgangireddy]

Hi @LorenzoF6 ,
Can you share your code or your config yaml so that I can try to understand more or reproduce your error.
Please also provide some details - OS, GPU, etc on which you are running your code.

[LorenzoF6]

Hi, the code is very simple

eng_patchcore = Engine(default_root_dir = 'risultati')
md_patchcore = Patchcore(evaluator = ev_SUOLE)
datamodule_SUOLE_train = Folder(
        name = "SUOLE", 
        root = dataset_path_SUOLE,
        normal_dir = "train/good",
        normal_test_dir = "test/good",
        train_batch_size = 2,
        num_workers = 1
        abnormal_dir = "test/0",
        mask_dir = "mask/0", 
)
datamodule_SUOLE_train.setup()
eng_patchcore.fit(model = md_patchcore, datamodule = datamodule_SUOLE_train)

I'm running on a notebook installed on a docker with 8 GB of GPU.
It is possibile that the problem are the many images, so it is possible to reduce the number of trained images? Can i work on the model's parameters? Thanks

[samet-akcay]

@LorenzoF6, batch size is not effective to avoid oom issue when training patchcore. The reason for oom is patchcore is a memory bank-based model where it combines all the features into a memory bank. If your dataset is large, the bigger memory bank, hence more susceptible to oom.

The following are the parameters you could play with to reduce the dimension of the embedding you are extracting and applying dimensionality reduction

anomalib/src/anomalib/models/image/patchcore/lightning_model.py

Lines 138 to 142 in ee5d986

	backbone: str = "wide_resnet50_2",
	layers: Sequence[str] = ("layer2", "layer3"),
	pre_trained: bool = True,
	coreset_sampling_ratio: float = 0.1,
	num_neighbors: int = 9,

Since this is not a bug, I'm moving this to a Q&A.

Note, however, is that optimizing patchcore for a more efficient memory bank approach is on our roadmap.

[LorenzoF6]

Thansk, so i suggest that if i have limited GPU resources, and i think that modify the parameters not change a lot if i want to use my entire dataset, a possible strategy is to subsample it, reducing to 200/300 images and than test it, correct?
Thansk

[waschsalz]

Hello @LorenzoF6,

if you look into benchmark datasets like MVTec (and others) you will realize that many classes (categories as they are called in anomalib) consist of 200-300 training images. So subsampling is definitely appropriate and in that case mandatory, especially if you want to use memory bank-based models like Padim or Patchcore.
If you aren't for some reason limited to this type of models, you should try out reconstructing-based models such as EfficientAD or UniNet.

Note that you should ask yourself while acquiring data (images) how strongly they differ from each other. If you really have 1000 super-identical images, then you can easily cut that down to 100, 150, 200 images in my opinion (trial & error). If you have bigger differences throughout your training images, then you should use more images but you need to make sure to have a good mix and equal distribution of various images.

Are your images identical in terms of lighting and camera position (+zoom and so on) ?

[rajeshgangireddy]

Hi,
We recently did some memory optimisation for Patchcore :
#2813

This should reduce the chance of oom significantly.