Does _setup_transform() in Engine.fit use the train transforms properly?

[CarlosNacher]

Discussed in #2100

^{Originally posted by CarlosNacher May 31, 2024}
Hi,

I was looking the Engine.fit() method and I realised that maybe we are incorrectly using eval_transform for train instead of train_transform; or maybe I have misunderstood the code structure. I mean:

If you look into the fit() method of the Engine (engine.py line 441 ) the method _setup_transform() is called. And that method (line 311 of engine.py) does:

# get transform
if datamodule and datamodule.transform:
    # a transform passed explicitly to the datamodule takes precedence
    transform = datamodule.transform

And if you go to datamodule (datamodule.py line 232 in the transform property):

def transform(self) -> Transform:
    """Property that returns the user-specified transform for the datamodule, if any.

    This property is accessed by the engine to set the transform for the model. The eval_transform takes precedence
    over the train_transform, because the transform that we store in the model is the one that should be used during
    inference.
    """
    if self._eval_transform:
        return self._eval_transform
    if self._train_transform:
        return self._train_transform
    return None

it seems that even in fit method, the eval_transform takes precedence over train_transform. But if you are fitting, you want train_transform, not eval one.

Am I misunderstanding anything?

Thank you so much in advance!

[alexriedel1]

Whats happening in _setup_transform is setting the evaluation transform as the models default image transformation method. This is intended because during inference you want to use the eval transform only. The eval transform is stored to the model and saved in the model export file.

During training and validation the images are not transformed by the model transforms, but by the transforms that are defined in the respective datamodules.

[samet-akcay]

In addition to what @alexriedel1 described, i could add the following: if the user does not provide any custom transforms, the datamodule will use the transforms pre-defined by the model. These are the default ones that are reported in their respective papers.

If the user wants to provide custom transforms, the datamodule takes those into account. At the end of the training they will be wrapped within the exported model. The idea behind is that a user should be able to pass a raw image to the inference model without requiring any additional pre-processing stages. All of this will be handled by the inference model.

[samet-akcay]

I'm moving this to Q&A, from which we could continue to discuss.

[CarlosNacher]

Ohh, now I see it. I have dived deep into the code and I have understood, let me sum up my notes (in case someone might find it helpful):

• train batches are gotten from datamodule's train_dataloader, which uses train_data which is an AnomalibDataset class to which train_transform is passed. So during training always it is used train_transform.
  • And what is train_transform? If you look at train_transform AnomalibDatamodules property, you'll see that it is (preference from left to right): The config passed train_transform > (if not passed) the model_transform> (if not model transform) just resize to passed image_size in config > (if not image_size passed) None.
• In the same way, eval_transform (used for val and test dataloaders transforms) is: passed eval_transform in config > (if not passed) the model_transform> (if not model transform) just resize to passed image_size in config > (if not image_size passed) None.
• And what is model_transform that is in the train_transform and eval_transform precedence choice?: If you look at Engine's _setup_transform method (which is called when fit is called for example) you'll see that it is: datamodule's transform == eval_transform > (if eval_transform is None) dataloaders transform > (if not dataoladers passed) if checkpoint is passed, the transforms that that model had > (if not passed ckpt) model default transform (see each model class configure_transform method) > None. And that model_transform is used for the exported inference model.

Thank you so much for you helpful response!

Cheers,
N

[samet-akcay]

This is a great summary, Thanks @CarlosNacher!

[samet-akcay]

For those who are interested in some extra explanation and examples, here is a section from the documentation.
https://anomalib.readthedocs.io/en/v1.1.0/markdown/guides/how_to/data/transforms.html