Training on Custom dataset fails

[pmudgal-Intel]

Describe the bug

• Training PaDiM on custom dataset fails abruptly. Shows no errors but gives "Killed" message.
• Attaching the screenshots with error and config modification.
• Dataset - https://www.kaggle.com/datasets/arunrk7/surface-crack-detection

Expected behavior

• A clear and concise description of what you expected to happen.
• Ideally, the training should conclude successfully.

Screenshots

The config file is changed by following instructions on github.

Hardware and Software Configuration

• OS: [Ubuntu 20.04]
• NVIDIA Driver Version [470.141.03]
• CUDA Version [e.g. 10.2]
• CUDNN Version [e.g. V10.2.89]
• OpenVINO Version [Optional e.g. v2022.1.0]

[ashwinvaidya17]

Can you run again with htop open on a separate terminal? My suspicion is that the RAM usage is high. Linux kills jobs that take up the whole ram. What does sudo dmesg after the job is killed?

[abelBEDOYA]

Can you run again with htop open on a separate terminal? My suspicion is that the RAM usage is high. Linux kills jobs that take up the whole ram. What does sudo dmesg after the job is killed?

In my case, similarly, the training is killed. It is able to complete only one epoch.

Trying sudo dmesg , it show up this message:

I've tried reducing the batch sizes but still

[ashwinvaidya17]

Batch size is not the issue here then. If you look at this line https://github.com/openvinotoolkit/anomalib/blob/c1f51a6ccdb7cb26cd201a846f8049ac11b4e5cc/anomalib/models/padim/lightning_model.py#L78 it stores the embeddings for the entire training data with append. You can try playing around with the embedding dimension in the torch model here.https://github.com/openvinotoolkit/anomalib/blob/c1f51a6ccdb7cb26cd201a846f8049ac11b4e5cc/anomalib/models/padim/torch_model.py#L18 I am not sure if there are other ways to reduce memory consumption. @samet-akcay or @djdameln do you guys have any ideas?

[ngadhvi]

@pmudgal-Intel As you know, unlike traditional deep learning model training, the training with Anomalib PaDiM model happens in just one epoch (it's actually fitting a latent space than training for PaDiM). This will try to fit your whole data set into the memory. So, if your data set size increases by the total RAM you have, the training crashes.
Some suggestions:

1. Terminate all processes that consume the RAM before you start training.
2. Resize isn't an option for you as you are training with 256x256 images, try a GPU with 16 GB RAM.
3. Set the num_workers to 1 or 2. You will have to know how many cores your CPU has for this hyperparameter.
4. I saw 40k images in the dataset you are using. I would trim my dataset.

Hope this helps.

[abelBEDOYA]

@pmudgal-Intel As you know, unlike traditional deep learning model training, the training with Anomalib PaDiM model happens in just one epoch (it's actually fitting a latent space than training for PaDiM). This will try to fit your whole data set into the memory. So, if your data set size increases by the total RAM you have, the training crashes. Some suggestions:

1. Terminate all processes that consume the RAM before you start training.

2. Resize isn't an option for you as you are training with 256x256 images, try a GPU with 16 GB RAM.

3. Set the num_workers to 1 or 2. You will have to know how many cores your CPU has for this hyperparameter.

4. I saw 40k images in the dataset you are using. I would trim my dataset.

Hope this helps.

Thanks for the info, it is good to know that. Do the other models available in Anomalib work differently? That is, can they carry out an arbitrary number of epochs? (PatchCore, CFlow, GANomaly, ...)

[yonafalinie]

Hi, similiar in my case, GANomaly, Reverse Distillation, Fastflow and STFPM works best for me with that large number of images in a dataset, and can run with the default number of epochs given in the config file. I also modify the default resizing option, from 256 to 128 and as low as 64, which may hurt the performance in the end.

[pmudgal-Intel]

Training on custom dataset works after trimming the dataset.