Saving The Best Weight

[GanAndreas]

Hi all, I finished my EfficientAD training on my custom dataset, and I realized that anomalib actually saved the latest weight, not the best weight.
Is there anyway to save the best weight from training?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[samet-akcay]

When you use anomalib's Engine without any special configuration, it automatically sets up a checkpoint system that looks like this:

ModelCheckpoint(
    dirpath=default_root_dir / "weights" / "lightning",
    filename="model",
    auto_insert_metric_name=False,
)

Most anomalib models automatically track how well they're doing by logging metrics like: pixel_AUROC, image_AUROC, pixel_F1Score and image_F1Score.

The system keeps an eye on these metrics and saves the model version that performs the best.

How to Control Checkpoint Behavior

If You Want the Best Model (Default)

from anomalib.callbacks import ModelCheckpoint

# This will save your best model based on validation AUROC
best_checkpoint = ModelCheckpoint(
    dirpath="weights/best",
    filename="best_model_{epoch:02d}_{pixel_AUROC:.4f}",
    monitor="pixel_AUROC",
    mode="max",
    save_top_k=1,
    save_last=False,
    verbose=True
)

If You Want the Last Model

# This will save only the last epoch
last_checkpoint = ModelCheckpoint(
    dirpath="weights/last",
    filename="last_model_{epoch:02d}",
    monitor=None,  # No monitoring
    save_top_k=0,  # Don't save best
    save_last=True,  # Save last checkpoint
    verbose=True
)

If You Want Both

# This gives you the best of both worlds
both_checkpoint = ModelCheckpoint(
    dirpath="weights/both",
    filename="model_{epoch:02d}_{pixel_AUROC:.4f}",
    monitor="pixel_AUROC",
    mode="max",
    save_top_k=3,  # Keep the top 3 best models
    save_last=True,  # Also save the last model as "last.ckpt"
    verbose=True
)

Quick Example

Here's a simple way to use it:

from anomalib.data import MVTecAD
from anomalib.engine import Engine
from anomalib.models import Draem

# This will automatically save the best weights
datamodule = MVTecAD()
model = Draem()
engine = Engine(max_epochs=10)

engine.fit(datamodule=datamodule, model=model)

# You can find your best model here
best_path = engine.best_model_path
print(f"Best model saved at: {best_path}")

How to Check What's Happening

Want to make sure it's working as expected? Here's how you can check:

# After training
checkpoint_callback = engine.checkpoint_callback
if checkpoint_callback:
    print(f"Monitor: {checkpoint_callback.monitor}")
    print(f"Mode: {checkpoint_callback.mode}")
    print(f"Save top k: {checkpoint_callback.save_top_k}")
    print(f"Save last: {checkpoint_callback.save_last}")
    print(f"Best model path: {checkpoint_callback.best_model_path}")
    print(f"Best model score: {checkpoint_callback.best_model_score}")

1. If you set monitor=None, you'll get the last weights instead of the best ones
2. Make sure you use mode="max" for metrics like AUROC (higher is better)
3. Be careful with multiple checkpoint callbacks - they might confuse each other

Some Best Practices

1. Be explicit about what you're monitoring:

   ModelCheckpoint(monitor="pixel_AUROC", mode="max")

2. Pick the right metric for your task:

   • For pixel-level stuff: pixel_AUROC, pixel_F1Score
   • For image-level stuff: image_AUROC, image_F1Score

3. Save both best and last if you're not sure:

   ModelCheckpoint(
       monitor="pixel_AUROC",
       mode="max",
       save_top_k=1,
       save_last=True
   )

References

Check out these resources:

• PyTorch Lightning ModelCheckpoint Docs
• Anomalib ModelCheckpoint Code
• Anomalib Engine Code