Evaluator formed with torchmetrics metrics cannot be saved after model training #2753

Narc17 · 2025-06-06T04:09:32Z

Narc17
Jun 6, 2025

Describe the bug

Greetings again!

I am trying to utilize BinaryAccuracy from torchmetrics in test evaluator.

I used anomalib.metrics.create_anomalib_metric to wrap Metric class into AnomalibMetric class.

When Engine.trainer is trying to create checkpoint, it raised an error that the wrapped class cannot be pickled.

What should I do to avoid this error?

Thank you!

Dataset

MVTecAD

Model

PatchCore

Steps to reproduce the behavior

Create an Evaluator by:
- Evaluator(test_metrics=[anomalib.metrics.create_anomalib_metric(torchmetrics.classification.BinaryAccuracy)(fields=["pred_score", "gt_label"], prefix="image_")])
Initialize Patchcore with the evaluator created in step 1
Run model training with Engine.fit()
I will raise _pickle.PicklingError.

OS information

OS information:

OS: Ubuntu 24.04 (docker container)
Python version: 3.11.12
Anomalib version: 2.0.0
PyTorch version: 2.1.2
CUDA/cuDNN version: 12.6; driver version: 560.76
GPU models and configuration: 1x GeForce RTX 3070 Ti Laptop
Any other relevant information:
- Using a sub-dataset from MVTec dataset, the hazelnut dataset.
- Evaluator setup:
  - Evaluator(test_metrics=[anomalib.metrics.create_anomalib_metric(torchmetrics.classification.BinaryAccuracy)(fields=["pred_score", "gt_label"], prefix="image_")])

Expected behavior

Evaluate the model with BinaryAccuracy successfully.

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

No response

Configuration YAML

I did not utilize yaml configs.

Logs

File "/src/anomalib_test.py", line 553, in <module>
    engine.fit(datamodule=datamodule, model=lightning_model)
  File "/usr/local/lib/python3.11/dist-packages/anomalib/engine/engine.py", line 416, in fit
    self.trainer.fit(model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 538, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/call.py", line 47, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 574, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 981, in _run
    results = self._run_stage()
              ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 1025, in _run_stage
    self.fit_loop.run()
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/fit_loop.py", line 206, in run
    self.on_advance_end()
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/fit_loop.py", line 378, in on_advance_end
    call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=True)
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/call.py", line 218, in _call_callback_hooks
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 325, in on_train_epoch_end
    self._save_topk_checkpoint(trainer, monitor_candidates)
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 387, in _save_topk_checkpoint
    self._save_none_monitor_checkpoint(trainer, monitor_candidates)
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 715, in _save_none_monitor_checkpoint
    self._save_checkpoint(trainer, filepath)
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 390, in _save_checkpoint
    trainer.save_checkpoint(filepath, self.save_weights_only)
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 1365, in save_checkpoint
    self.strategy.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/strategies/strategy.py", line 490, in save_checkpoint
    self.checkpoint_io.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
  File "/usr/local/lib/python3.11/dist-packages/lightning/fabric/plugins/io/torch_io.py", line 58, in save_checkpoint
    _atomic_save(checkpoint, path)
  File "/usr/local/lib/python3.11/dist-packages/lightning/fabric/utilities/cloud_io.py", line 85, in _atomic_save
    torch.save(checkpoint, bytesbuffer)
  File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 619, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol, _disable_byteorder_record)
  File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 831, in _save
    pickler.dump(obj)
_pickle.PicklingError: Can't pickle <class 'abc.BinaryAccuracy'>: attribute lookup BinaryAccuracy on abc failed

Code of Conduct

I agree to follow this project's Code of Conduct

samet-akcay · 2025-06-06T04:32:36Z

samet-akcay
Jun 6, 2025
Maintainer

Thanks for reporting @Narc17. We'll investigate this

0 replies

samet-akcay · 2025-06-09T16:29:10Z

samet-akcay
Jun 9, 2025
Maintainer

@Narc17, I'm unable to reproduce this issue. The following script works on my side:

from torchmetrics.classification import BinaryAccuracy

from anomalib.data import MVTec
from anomalib.engine import Engine
from anomalib.metrics import Evaluator, create_anomalib_metric
from anomalib.models import Patchcore

# Create a datamodule
datamodule = MVTec()

# Create an Anomalib metric from a torchmetrics metric
Accuracy = create_anomalib_metric(BinaryAccuracy)
evaluator = Evaluator(
    test_metrics=[
        Accuracy(fields=["pred_score", "gt_label"], prefix="image_"),
    ],
)

# Create a model with the evaluator
model = Patchcore(evaluator=evaluator)

# Create an engine to train the model
engine = Engine()
engine.train(model=model, datamodule=datamodule)

Here are the terminal results when you run this:

❯ python debug_evaluator.py
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/sa/Projects/anomalib/debug_evaluator.py:9: DeprecationWarning: MVTec is deprecated and will be removed in a future version. Please use MVTecAD instead.
  datamodule = MVTec()
INFO:anomalib.models.components.base.anomalib_module:Initializing Patchcore model.
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/lightning/pytorch/utilities/parsing.py:209: Attribute 'evaluator' is an instance of `nn.Module` and is already saved during checkpointing. It is recommended to ignore them using `self.save_hyperparameters(ignore=['evaluator'])`.
INFO:timm.models._builder:Loading pretrained weights from Hugging Face hub (timm/wide_resnet50_2.racm_in1k)
INFO:timm.models._hub:[timm/wide_resnet50_2.racm_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
INFO:timm.models._builder:Missing keys (fc.weight, fc.bias) discovered while loading pretrained weights. This is expected if model is being adapted.
INFO:lightning_fabric.utilities.rank_zero:GPU available: True (cuda), used: True
INFO:lightning_fabric.utilities.rank_zero:TPU available: False, using: 0 TPU cores
INFO:lightning_fabric.utilities.rank_zero:HPU available: False, using: 0 HPUs
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/lightning/pytorch/utilities/parsing.py:45: Attribute 'evaluator' removed from hparams because it cannot be pickled. You can suppress this warning by setting `self.save_hyperparameters(ignore=['evaluator'])`.
INFO:lightning_fabric.utilities.rank_zero:You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
/home/sa/.cursor-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/_vendored/force_pydevd.py:18: UserWarning: incompatible copy of pydevd already imported:
 /home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/pydevd_plugins/extensions/pydevd_plugin_omegaconf.py
  warnings.warn(msg + ':\n {}'.format('\n  '.join(_unvendored)))
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/sa/Projects/anomalib/debug_evaluator.py:9: DeprecationWarning: MVTec is deprecated and will be removed in a future version. Please use MVTecAD instead.
  datamodule = MVTec()
INFO:anomalib.models.components.base.anomalib_module:Initializing Patchcore model.
INFO:timm.models._builder:Loading pretrained weights from Hugging Face hub (timm/wide_resnet50_2.racm_in1k)
INFO:timm.models._hub:[timm/wide_resnet50_2.racm_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
INFO:timm.models._builder:Missing keys (fc.weight, fc.bias) discovered while loading pretrained weights. This is expected if model is being adapted.
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
INFO:lightning_fabric.utilities.rank_zero:----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------

INFO:anomalib.data.datamodules.image.mvtecad:Found the dataset.
WARNING:anomalib.metrics.evaluator:Number of devices is greater than 1, setting compute_on_cpu to False.
WARNING:anomalib.metrics.evaluator:Number of devices is greater than 1, setting compute_on_cpu to False.
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/lightning/pytorch/core/optimizer.py:183: `LightningModule.configure_optimizers` returned `None`, this fit will run with no optimizer

  | Name           | Type           | Params | Mode 
----------------------------------------------------------
0 | pre_processor  | PreProcessor   | 0      | train
1 | post_processor | PostProcessor  | 0      | train
2 | evaluator      | Evaluator      | 0      | train
3 | model          | PatchcoreModel | 24.9 M | train
----------------------------------------------------------
24.9 M    Trainable params
0         Non-trainable params
24.9 M    Total params
99.450    Total estimated model params size (MB)
16        Modules in train mode
174       Modules in eval mode
Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:05<00:00,  0.72it/s]INFO:anomalib.models.image.patchcore.lightning_model:Aggregating the embedding extracted from the training set.
INFO:anomalib.models.image.patchcore.lightning_model:Applying core-set subsampling to get the embedding.
                                                                                                                                                                               INFO:anomalib.models.image.patchcore.lightning_model:Aggregating the embedding extracted from the training set.                                            | 0/? [00:00<?, ?it/s]
INFO:anomalib.models.image.patchcore.lightning_model:Applying core-set subsampling to get the embedding.
Selecting Coreset Indices.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 10752/10752 [00:09<00:00, 1099.51it/s]
Selecting Coreset Indices.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 10752/10752 [00:10<00:00, 1070.63it/s]
Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:28<00:00,  0.14it/s]INFO:lightning_fabric.utilities.rank_zero:`Trainer.fit` stopped: `max_epochs=1` reached.                                                                                        
Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:29<00:00,  0.14it/s]
INFO:anomalib.callbacks.timer:Training took 31.32 seconds
INFO:lightning_fabric.utilities.rank_zero:The following callbacks returned in `LightningModule.configure_callbacks` will override existing callbacks passed to Trainer: Evaluator, ImageVisualizer, PostProcessor, PreProcessor
INFO:anomalib.callbacks.timer:Training took 31.33 seconds
INFO:anomalib.data.datamodules.image.mvtecad:Found the dataset.
WARNING:anomalib.metrics.evaluator:Number of devices is greater than 1, setting compute_on_cpu to False.
WARNING:anomalib.metrics.evaluator:Number of devices is greater than 1, setting compute_on_cpu to False.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:216: Using `DistributedSampler` with the dataloaders. During `trainer.test()`, it is recommended to use `Trainer(devices=1, num_nodes=1)` to ensure each sample/batch gets evaluated exactly once. Otherwise, multi-device settings use `DistributedSampler` that replicates some samples to make sure all devices have same batch size in case of uneven inputs.
Testing DataLoader 0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00,  0.86it/s]INFO:anomalib.callbacks.timer:Testing took 6.876540184020996 seconds
Throughput (batch_size=32) : 12.070023264441463 FPS
INFO:anomalib.callbacks.timer:Testing took 6.854757308959961 seconds
Throughput (batch_size=32) : 12.10837908024977 FPS
Testing DataLoader 0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  0.67it/s]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│   image_BinaryAccuracy    │     0.988095223903656     │
└───────────────────────────┴───────────────────────────┘

Here are the package versions on which I tested this

torch==2.6.0
torch-tb-profiler==0.4.3
torchmetrics==1.7.0
torchvision==0.21.0

0 replies

samet-akcay · 2025-06-10T08:56:06Z

samet-akcay
Jun 10, 2025
Maintainer

I'm moving this to a Q&A. Feel free to convert it back to a bug report if you still think there is something wrong. Cheers!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Evaluator formed with torchmetrics metrics cannot be saved after model training #2753

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Evaluator formed with torchmetrics metrics cannot be saved after model training #2753

Uh oh!

Narc17 Jun 6, 2025

Describe the bug

Dataset

Model

Steps to reproduce the behavior

OS information

Expected behavior

Screenshots

Pip/GitHub

What version/branch did you use?

Configuration YAML

Logs

Code of Conduct

Replies: 3 comments

Uh oh!

samet-akcay Jun 6, 2025 Maintainer

Uh oh!

Uh oh!

samet-akcay Jun 9, 2025 Maintainer

Uh oh!

samet-akcay Jun 10, 2025 Maintainer

Narc17
Jun 6, 2025

samet-akcay
Jun 6, 2025
Maintainer

samet-akcay
Jun 9, 2025
Maintainer

samet-akcay
Jun 10, 2025
Maintainer