❓ <title>Can you please tell me the specific training parameters that are officially used to train the MVTecAD dataset using patchcore?

[FreemanTang]

Your Question

Can you please tell me the specific training parameters that are officially used to train the MVTecAD dataset using patchcore?

The results I get when I use this code to train are way off compared to the official results, Patchcore Results. I would like to know what is the reason for this?

from anomalib.data import MVTecAD
from anomalib.engine import Engine
from anomalib.models import Patchcore

from pathlib import Path
from anomalib.data import Folder

# 1. Basic Usage
# Initialize with default settings
model = Patchcore()

# 2. Custom Configuration
# Configure model parameters
model = Patchcore(
    backbone="wide_resnet50_2",  # Feature extraction backbone
    layers=["layer2", "layer3"],  # Layers to extract features from
    pre_trained=True,  # Use pretrained weights
    num_neighbors=9,  # Number of nearest neighbors
)

# 3. Training Pipeline
datamodule = MVTecAD(
    root="anomalib/datasets/MVTec",
    category="bottle",
    train_batch_size=16,
    eval_batch_size=16,  # Important for feature extraction
    num_workers=0,  # Number of workers for data loading
    seed=42,
)


# Initialize training engine with specific settings
engine = Engine(
    max_epochs=1,  # Patchcore typically needs only one epoch
    accelerator="auto",  # Automatically detect GPU/CPU
    devices=1,  # Number of devices to use
)

# Train the model
# engine.fit(
#     model=model,
#     datamodule=datamodule,
# )

engine.train(
    model=model,
    datamodule=datamodule,
)

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        image_AUROC        │            1.0            │
│       image_F1Score       │    0.9919999837875366     │
│        pixel_AUROC        │    0.9855615496635437     │
│       pixel_F1Score       │    0.7266571521759033     │
└───────────────────────────┴───────────────────────────┘

Forum Check

• I have searched the discussions forum for an answer to my question.

[github-actions[bot]]

Thanks for submitting this issue! It has been added to our triage queue. A maintainer will review it shortly.

[alexriedel1]

Hey, your results are not really off of these https://github.com/open-edge-platform/anomalib/blob/main/src/anomalib/models/image/patchcore/README.md#mvtec-ad-dataset

Only your Image F1 is 0.9919999837875366 compared to 1.0 but I wouldn't call it way off

[FreemanTang]

Thank you for your reply! I have one more question, I want to ask you，
Why there will be a difference in the results between using anomalib.data.Folder to read MVTecAD data for training and anomalib.data.MVTecAD to read training directly?

anomalib train --config config_Folder.yaml
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        image_AUROC        │            1.0            │
│       image_F1Score       │    0.9743589758872986     │
│        pixel_AUROC        │    0.9921541213989258     │
│       pixel_F1Score       │    0.8096356391906738     │
└───────────────────────────┴───────────────────────────┘

anomalib train --config config_MVTecAD.yaml
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        image_AUROC        │            1.0            │
│       image_F1Score       │    0.9919999837875366     │
│        pixel_AUROC        │    0.9855770468711853     │
│       pixel_F1Score       │    0.7263702154159546     │
└───────────────────────────┴───────────────────────────┘

# config_Folder.yaml
# anomalib==2.2.0
seed_everything: true
trainer:
  accelerator: auto
  strategy: auto
  devices: auto
  num_nodes: 1
  precision: null
  logger: null
  callbacks: null
  fast_dev_run: false
  max_epochs: null
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: null
  limit_val_batches: null
  limit_test_batches: null
  limit_predict_batches: null
  overfit_batches: 0.0
  val_check_interval: null
  check_val_every_n_epoch: 1
  num_sanity_val_steps: null
  log_every_n_steps: null
  enable_checkpointing: null
  enable_progress_bar: null
  enable_model_summary: null
  accumulate_grad_batches: 1
  gradient_clip_val: null
  gradient_clip_algorithm: null
  deterministic: null
  benchmark: null
  inference_mode: true
  use_distributed_sampler: true
  profiler: null
  detect_anomaly: false
  barebones: false
  plugins: null
  sync_batchnorm: false
  reload_dataloaders_every_n_epochs: 0
  enable_autolog_hparams: true
  model_registry: null
optimizer: null
lr_scheduler: null
model:
  class_path: anomalib.models.Patchcore
  init_args:
    backbone: wide_resnet50_2
    layers:
    - layer2
    - layer3
    pre_trained: true
    coreset_sampling_ratio: 0.1
    num_neighbors: 9
    pre_processor: true
    post_processor: true
    evaluator: true
    visualizer: true
data:
  class_path: anomalib.data.Folder
  init_args:
    name: bottle
    normal_dir: train/good
    root: datasets/MVTecAD/bottle
    abnormal_dir: test/broken_large
    normal_test_dir: test/good
    mask_dir: ground_truth/broken_large
    normal_split_ratio: 0.2
    extensions:
    - .png
    train_batch_size: 16
    eval_batch_size: 16
    num_workers: 8
    train_augmentations: null
    val_augmentations: null
    test_augmentations: null
    augmentations: null
    test_split_mode: FROM_DIR
    test_split_ratio: 0.2
    val_split_mode: SAME_AS_TEST
    val_split_ratio: 0.5
    seed: null
logging:
  log_graph: false
default_root_dir: results
ckpt_path: null


# config_MVTecAD.yaml
# anomalib==2.2.0
seed_everything: true
trainer:
  accelerator: auto
  strategy: auto
  devices: auto
  num_nodes: 1
  precision: null
  logger: null
  callbacks: null
  fast_dev_run: false
  max_epochs: null
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: null
  limit_val_batches: null
  limit_test_batches: null
  limit_predict_batches: null
  overfit_batches: 0.0
  val_check_interval: null
  check_val_every_n_epoch: 1
  num_sanity_val_steps: null
  log_every_n_steps: null
  enable_checkpointing: null
  enable_progress_bar: null
  enable_model_summary: null
  accumulate_grad_batches: 1
  gradient_clip_val: null
  gradient_clip_algorithm: null
  deterministic: null
  benchmark: null
  inference_mode: true
  use_distributed_sampler: true
  profiler: null
  detect_anomaly: false
  barebones: false
  plugins: null
  sync_batchnorm: false
  reload_dataloaders_every_n_epochs: 0
  enable_autolog_hparams: true
  model_registry: null
optimizer: null
lr_scheduler: null
model:
  class_path: anomalib.models.Patchcore
  init_args:
    backbone: wide_resnet50_2
    layers:
    - layer2
    - layer3
    pre_trained: true
    coreset_sampling_ratio: 0.1
    num_neighbors: 9
    pre_processor: true
    post_processor: true
    evaluator: true
    visualizer: true
data:
  class_path: anomalib.data.MVTecAD
  init_args:
    root: datasets\MVTecAD
    category: bottle
    train_batch_size: 16
    eval_batch_size: 16
    num_workers: 8
    train_augmentations: null
    val_augmentations: null
    test_augmentations: null
    augmentations: null
    test_split_mode: FROM_DIR
    test_split_ratio: 0.2
    val_split_mode: SAME_AS_TEST
    val_split_ratio: 0.5
    seed: null
logging:
  log_graph: false
default_root_dir: results
ckpt_path: null

[alexriedel1]

Thats probably because with the folder dataset, you are only testing on one type of defects

abnormal_dir: test/broken_large
normal_test_dir: test/good
mask_dir: ground_truth/broken_large