❓ Device inconsistency when utilizing AUPRO metric

[Narc17]

Describe the bug

Greetings!

I faced a tensor device inconsistent issue while applying AUPRO in Patchcore validation/testing. The model training was did utilize cuda and was succeeded. However, it raised error about two tensors on different devices in validation or testing. This issue only occurs on AUPRO metric. How can I slove this issue?

Thanks!

Dataset

MVTecAD

Model

PatchCore

Steps to reproduce the behavior

1. Create an Evaluator with AUPRO image and pixel level metrics.
   • Evaluator(test_metrics=[AUPRO(fields=["pred_score", "gt_label"], prefix="image_"), AUPRO(fields=["anomaly_map", "gt_mask"], prefix="pixel_", strict=False) ])
2. Initialize Patchcore with evaluator={evaluator created with step 1}
3. Initialize Engine and do Engine.fit() and Engine.test()
4. It will raise the error above during processing Engine.test()

OS information

OS information:

• OS: Ubuntu 24.04 (docker container)
• Python version: 3.11.12
• Anomalib version: 2.0.0
• PyTorch version: 2.1.2
• CUDA/cuDNN version: 12.6; driver version: 560.76
• GPU models and configuration: 1x GeForce RTX 3070 Ti Laptop
• Any other relevant information:
  • Using a sub-dataset from MVTec dataset, the hazelnut dataset.
  • Evaluator setup:
    • Evaluator(test_metrics=[AUPRO(fields=["pred_score", "gt_label"], prefix="image_"), AUPRO(fields=["anomaly_map", "gt_mask"], prefix="pixel_", strict=False) ])

Expected behavior

AUPRO metric could be handled GPU(cuda) computations.

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

2.0.0

Configuration YAML

Did not utilize yaml configs.

Logs

Traceback (most recent call last):
  File "/src/anomalib_script.py", line 594, in <module>
    test_engine.test(datamodule=test_datamodule, model=test_lightning_model)
  File "/usr/local/lib/python3.11/dist-packages/anomalib/engine/engine.py", line 558, in test
    return self.trainer.test(model, dataloaders, ckpt_path, verbose, datamodule)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 748, in test
    return call._call_and_handle_interrupt(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/call.py", line 47, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 788, in _test_impl
    results = self._run(model, ckpt_path=ckpt_path)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 981, in _run
    results = self._run_stage()
              ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 1018, in _run_stage
    return self._evaluation_loop.run()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/utilities.py", line 178, in _decorator
    return loop_run(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/evaluation_loop.py", line 142, in run
    return self.on_run_end()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/evaluation_loop.py", line 254, in on_run_end
    self._on_evaluation_epoch_end()
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/evaluation_loop.py", line 336, in _on_evaluation_epoch_end
    trainer._logger_connector.on_epoch_end()
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 195, in on_epoch_end
    metrics = self.metrics
              ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 234, in metrics
    return self.trainer._results.metrics(on_step)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 473, in metrics
    value = self._get_cache(result_metric, on_step)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 437, in _get_cache
    result_metric.compute()
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 288, in wrapped_func
    self._computed = compute(*args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 253, in compute
    return self.value.compute()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/torchmetrics/metric.py", line 699, in wrapped_func
    value = _squeeze_if_scalar(compute(*args, **kwargs))
                               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anomalib/metrics/base.py", line 191, in compute
    return super().compute()  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anomalib/metrics/aupro.py", line 307, in compute
    fpr, tpr = self._compute()
               ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anomalib/metrics/aupro.py", line 299, in _compute
    return self.compute_pro(cca=cca, target=target, preds=preds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anomalib/metrics/aupro.py", line 219, in compute_pro
    output_size = torch.where(fpr <= self.fpr_limit)[0].size(0)
                              ^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Code of Conduct

• I agree to follow this project's Code of Conduct

[samet-akcay]

@Narc17, I tried to reproduce this issue, here is my findings:

i see that you are trying to create an AUPRO as an image metric. This won't work as it is a pixel-level metric. If you, therefore, remove the image-level metric that you set in Evaluator it should work.

Here is the code I tried, which achieved 0.945 AUPRO score.

from anomalib.data import MVTec
from anomalib.engine import Engine
from anomalib.metrics import AUPRO, Evaluator
from anomalib.models import Patchcore

evaluator = Evaluator(
    test_metrics=[
        # NOTE: AUPRO only works with pixel-level metrics.
        # AUPRO(fields=["pred_score", "gt_label"], prefix="image_"),
        AUPRO(fields=["anomaly_map", "gt_mask"], prefix="pixel_", strict=False),
    ],
)
datamodule = MVTec()
model = Patchcore(evaluator=evaluator)
engine = Engine()

engine.fit(model=model, datamodule=datamodule)
engine.test(model=model, datamodule=datamodule)

Here is the terminal output:

❯ python debug_aupro.py
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/sa/Projects/anomalib/debug_aupro.py:13: DeprecationWarning: MVTec is deprecated and will be removed in a future version. Please use MVTecAD instead.
  datamodule = MVTec()
INFO:anomalib.models.components.base.anomalib_module:Initializing Patchcore model.
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/lightning/pytorch/utilities/parsing.py:209: Attribute 'evaluator' is an instance of `nn.Module` and is already saved during checkpointing. It is recommended to ignore them using `self.save_hyperparameters(ignore=['evaluator'])`.
INFO:timm.models._builder:Loading pretrained weights from Hugging Face hub (timm/wide_resnet50_2.racm_in1k)
INFO:timm.models._hub:[timm/wide_resnet50_2.racm_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
INFO:timm.models._builder:Missing keys (fc.weight, fc.bias) discovered while loading pretrained weights. This is expected if model is being adapted.
INFO:lightning_fabric.utilities.rank_zero:GPU available: True (cuda), used: True
INFO:lightning_fabric.utilities.rank_zero:TPU available: False, using: 0 TPU cores
INFO:lightning_fabric.utilities.rank_zero:HPU available: False, using: 0 HPUs
INFO:lightning_fabric.utilities.rank_zero:You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
/home/sa/.cursor-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/_vendored/force_pydevd.py:18: UserWarning: incompatible copy of pydevd already imported:
 /home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/pydevd_plugins/extensions/pydevd_plugin_omegaconf.py
  warnings.warn(msg + ':\n {}'.format('\n  '.join(_unvendored)))
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/sa/Projects/anomalib/debug_aupro.py:13: DeprecationWarning: MVTec is deprecated and will be removed in a future version. Please use MVTecAD instead.
  datamodule = MVTec()
INFO:anomalib.models.components.base.anomalib_module:Initializing Patchcore model.
INFO:timm.models._builder:Loading pretrained weights from Hugging Face hub (timm/wide_resnet50_2.racm_in1k)
INFO:timm.models._hub:[timm/wide_resnet50_2.racm_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
INFO:timm.models._builder:Missing keys (fc.weight, fc.bias) discovered while loading pretrained weights. This is expected if model is being adapted.
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
INFO:lightning_fabric.utilities.rank_zero:----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------

INFO:anomalib.data.datamodules.image.mvtecad:Found the dataset.
WARNING:anomalib.metrics.evaluator:Number of devices is greater than 1, setting compute_on_cpu to False.
WARNING:anomalib.metrics.evaluator:Number of devices is greater than 1, setting compute_on_cpu to False.
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/lightning/pytorch/core/optimizer.py:183: `LightningModule.configure_optimizers` returned `None`, this fit will run with no optimizer

  | Name           | Type           | Params | Mode 
----------------------------------------------------------
0 | pre_processor  | PreProcessor   | 0      | train
1 | post_processor | PostProcessor  | 0      | train
2 | evaluator      | Evaluator      | 0      | train
3 | model          | PatchcoreModel | 24.9 M | train
----------------------------------------------------------
24.9 M    Trainable params
0         Non-trainable params
24.9 M    Total params
99.450    Total estimated model params size (MB)
16        Modules in train mode
174       Modules in eval mode
Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:05<00:00,  0.78it/sINFO:anomalib.models.image.patchcore.lightning_model:Aggregating the embedding extracted from the training set.                                            | 0/? [00:00<?, ?it/s]
INFO:anomalib.models.image.patchcore.lightning_model:Applying core-set subsampling to get the embedding.
INFO:anomalib.models.image.patchcore.lightning_model:Aggregating the embedding extracted from the training set.
INFO:anomalib.models.image.patchcore.lightning_model:Applying core-set subsampling to get the embedding.
Selecting Coreset Indices.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 10752/10752 [00:09<00:00, 1101.15it/s]
Selecting Coreset Indices.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 10752/10752 [00:10<00:00, 1074.65it/s]
Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:28<00:00,  0.14it/s]INFO:lightning_fabric.utilities.rank_zero:`Trainer.fit` stopped: `max_epochs=1` reached.                                                                                        
Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:32<00:00,  0.12it/s]
INFO:anomalib.callbacks.timer:Training took 34.07 seconds
INFO:anomalib.callbacks.timer:Training took 34.11 seconds
INFO:lightning_fabric.utilities.rank_zero:The following callbacks returned in `LightningModule.configure_callbacks` will override existing callbacks passed to Trainer: Evaluator, ImageVisualizer, PostProcessor, PreProcessor
INFO:anomalib.data.datamodules.image.mvtecad:Found the dataset.
WARNING:anomalib.metrics.evaluator:Number of devices is greater than 1, setting compute_on_cpu to False.
WARNING:anomalib.metrics.evaluator:Number of devices is greater than 1, setting compute_on_cpu to False.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
/home/sa/Projects/anomalib/.venv/lib/python3.11/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:216: Using `DistributedSampler` with the dataloaders. During `trainer.test()`, it is recommended to use `Trainer(devices=1, num_nodes=1)` to ensure each sample/batch gets evaluated exactly once. Otherwise, multi-device settings use `DistributedSampler` that replicates some samples to make sure all devices have same batch size in case of uneven inputs.
Testing DataLoader 0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00,  0.84it/s]INFO:anomalib.callbacks.timer:Testing took 7.9087910652160645 seconds
Throughput (batch_size=32) : 10.494650739358288 FPS
INFO:anomalib.callbacks.timer:Testing took 7.911909103393555 seconds
Throughput (batch_size=32) : 10.490514857457079 FPS
Testing DataLoader 0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:04<00:00,  0.50it/s]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        pixel_AUPRO        │     0.945563793182373     │
└───────────────────────────┴───────────────────────────┘

[samet-akcay]

I'm moving this to Q&A now as it does not seem to be an issue. Let's continue the discussion there.

[sonhm3029]

I found that the issue is in this:

Line 219. I print out and the fpr is in "cpu" but the self.fpr_limit is in "cuda"

@samet-akcay

[samet-akcay]

Nice find @sonhm3029! Would you be interested in creating a PR to become a contributor? Or would you prefer us fixing it?

[sonhm3029]

@samet-akcay Okay I will create a PR