What happens when val_split_mode = FROM_TRAIN?

[ZenoVolt]

Describe the bug

datamodule = Folder(
        name=os.path.basename(datasetFolder),
        root=datasetFolder,
        normal_dir="normal",
        abnormal_dir="abnormal", # Let Anomalib infer all subfolders = defect types
        normal_test_dir="normal_test",
        mask_dir="mask",
        test_split_mode=TestSplitMode.FROM_DIR,
        val_split_mode=ValSplitMode.FROM_TEST # using FROM_TRAIN cause problem???
    )

using val_split_mode = FROM_TRAIN seems to work incorrectly, when tested with MVTecAD dataset the heatmaps all show up as red.

Dataset

N/A

Model

N/A

Steps to reproduce the behavior

copy data into the expected folder structure
use ValSplitMode.FROM_TRAIN

def benchmarkDataset(report: AnomalyBenchmarkReport, datasetFolder: str):
    from anomalib.data import MVTecAD
    from anomalib.visualization import ImageVisualizer
    from anomalib.models import Patchcore
    from anomalib.engine import Engine
    from anomalib.data import Folder
    from anomalib.data.utils import (
        DirType,
        LabelName,
        Split,
        TestSplitMode,
        ValSplitMode,
    )

    # Create visualizer with default settings
    visualizer = ImageVisualizer()
    model = Patchcore(
        visualizer=visualizer
    )

    engine = Engine()
    
    datamodule = Folder(
            name=os.path.basename(datasetFolder),
            root=datasetFolder,
            normal_dir="normal",
            abnormal_dir="abnormal", # Let Anomalib infer all subfolders = defect types
            normal_test_dir="normal_test",
            mask_dir="mask",
            test_split_mode=TestSplitMode.FROM_DIR,
            val_split_mode=ValSplitMode.FROM_TEST # using FROM_TRAIN cause problem???
        )

    datamodule.setup()

    engine.fit(datamodule=datamodule, model=model)
    results = engine.test(datamodule=datamodule, model=model)
    #print(results)

    report.add_result_slide(model.__class__.__name__, os.path.basename(datasetFolder), results)
    report.addImageSlidesFromSubfolders("Normal Test", "results/Patchcore/" + datamodule.name + "/latest/images/normal_test", 10)
    report.addImageSlidesFromSubfolders("Abnormal Test", "results/Patchcore/" + datamodule.name + "/latest/images/abnormal", 10)

OS information

OS information:

• OS: [e.g. Ubuntu 20.04]
• Python version: [e.g. 3.10.0]
• Anomalib version: [e.g. 0.3.6]
• PyTorch version: [e.g. 1.9.0]
• CUDA/cuDNN version: [e.g. 11.1]
• GPU models and configuration: [e.g. 2x GeForce RTX 3090]
• Any other relevant information: [e.g. I'm using a custom dataset]

Expected behavior

Heat maps should not be red

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

No response

Configuration YAML

n/a

Logs

n/a

Code of Conduct

• I agree to follow this project's Code of Conduct

[samet-akcay]

@keithtangtw, I don't think this is a bug. Training set does not contain abnormal images. if you create your validation set from training set, you would not have any abnormal images in your validation set. If you don't have abnormal images in your validation set, anomalib cannot compute normalization/threshold value accurately. Since your values are not accurate, you have incorrect heatmaps.

[aCuria]

Thanks! this makes sense.