Training from custom dataloaders / torch dataset

[luizfelippesr]

Anomalib works natively with customized lighting DataModule objects, with the possibility creating those from files on the local file system (the FolderDataset) . Is there a recommended way of using a bespoke torch dataset/dataloader instead?

I have a particular use-case where the part of the data can be stored on other devices and can be retrieved on-the-fly (and possibly cached) by an adaptor written as a torch dataset. Also, I often prefer pre-splitting my data into train/val/test and storing separately to avoid the possibility of later contamination (and also allowing things like extending only the training set with a few extra examples).

Since the training engine uses lightning, it should be possible to simply using engine.fit(..., train_dataloaders=my_dataloader). However, this appears to give different results.

For example, if I take the example run:

# Import the required modules
from anomalib.data import MVTec
from anomalib.models import Patchcore
from anomalib.engine import Engine

# Initialize the datamodule, model and engine
datamodule = MVTec()
model = Patchcore()
engine = Engine()

# Train the model
engine.fit(datamodule=datamodule, model=model)

it finishes quickly in my system (using anomalib 1.1.0).

If modify in the following way (which should be equivalent):

# Import the required modules
from anomalib.data import MVTec
from anomalib.models import Patchcore
from anomalib.engine import Engine

# Initialize the datamodule, model and engine
datamodule = MVTec()
model = Patchcore()
engine = Engine()

# Prepares the data
datamodule.setup()

# Train the model
engine.fit(
    model=model,
    train_dataloaders=datamodule.train_dataloader(),
    val_dataloaders=datamodule.val_dataloader(),
)

I quickly run out of memory.

Any ideas?

[samet-akcay]

This might be related to this #2254

[luizfelippesr]

The workaround mentioned in #2254 bug report does not seem to help: the dataloader.dataset.transform is already None and the model seems to have the correct transform configured.

BTW, @samet-akcay should I re-write my question as a bug report?

[luizfelippesr]

Related: #2034

[blaz-r]

This is quite unusual, and I'm not sure what would be causing this. Are you running out of RAM or GPU RAM?