[Bug]: Some bugs about Padim and PatchCore (or maybe it's just a problem I'm having)

[laogonggong847]

Describe the bug

There are three problems in this submission, all of which I encountered while training my own dataset：

1: padim

When I use padim, I set padim's backbone to resnet18, and the program runs correctly and gets the results I want at this time.

I looked at the padim README.md and found that Padim performs even better when Backbone is Wide ResNet-50. So I modified the corresponding config.yaml of padim. But I only modified the backbone. as follows:

backbone: resnet18 ——> backbone: wide_resnet50_2

I have downloaded the corresponding pre-training file correctly. After modifying it to Wide ResNet-50, padim will report an error directly after a period of training (or is it when the training is completed and validated? Please see my screenshot below for details) and it reports an error directly. I don't think this should be the case. When I use other models (e.g. fastflow, etc.), I just modify the backbone without modifying other parameters and the training is done correctly in the end.

Q1: So my first question is: why is this happening after modifying Padim's backbone network to Wide ResNet-50 and how should I fix it

---

---

2:patchcore

patchcore I got the onnx model correctly with the corresponding results when setting Backbone to Wide ResNet-50. But when I use OnnxRuntime to infer the model, it reports an error directly.

I can ensure that my code for inference using OnnxRuntime is correct, and I am using OnnxRuntime correctly to infer the Onnx model for some other methods provided by Anomalib. But only PatchCore is problematic, and its specific error message will be given later in the form of a screenshot.

Q2: So my second question is how should I properly use OnnxRuntime to reason about my PatchCore's corresponding Onnx model

---

---

3: How to calculate the confidence score

This is not a bug, just a question I have. I checked how the final confidence score is calculated in the source code section, but I'm still confused.

As you can see, in the Prediction corresponding result graph, normal(59%) is clearly shown in the top left corner:

I would like to know what I should do when I use OnnxRuntime to rewrite the inference part myself, instead of using one of the four inference programs provided by Anomalib。

Q3: So my third question is: when I write my own program with OnnxRuntime (GPU) reasoning, how should I get the "normal" or "abnormal" judgments and how should I calculate the corresponding confidence score. It would be great if you could provide me with a demo of this, because I am new to this, so it might be difficult to implement the program based on some hints.

---

---

Thank you very much for seeing this, the problem is relatively long delayed too long, I hope to get your guidance, thank you very much!

Dataset

Other (please specify in the text field below)

Model

PatchCore

Steps to reproduce the behavior

Pytorch : 1.12.1+Cu113
GPU: RTX2080Ti
onnx: 1.12.0
onnxruntime-gpu: 1.14.1
CUDA: 11.3
OS: win10
CPU: Intel Core i7-8700K

Expected behavior

I would appreciate suggestions and solutions for problem 1, problem 2 and problem 3. Thank you very much, because my native language is not English, I may not be accurate, I hope you will understand more, thank you again!

Screenshots

1: Specific information on training errors when Padim sets backbone to wide_ResNet50_2

I was concerned that the screenshot related to question 1 might not have all the information, so I printed out the log related to question 1. The screenshots related to question 2 and question 3 are clear, so I won't repeat them here

---

| Name | Type | Params

0 | image_threshold | AnomalyScoreThreshold | 0
1 | pixel_threshold | AnomalyScoreThreshold | 0
2 | model | PadimModel | 24.9 M
3 | image_metrics | AnomalibMetricCollection | 0
4 | pixel_metrics | AnomalibMetricCollection | 0
5 | normalization_metrics | MinMax | 0

24.9 M Trainable params
0 Non-trainable params
24.9 M Total params
99.450 Total estimated model params size (MB)
Epoch 0: 6%|▋ | 1/16 [00:18<04:31, 18.09s/it, loss=nan]C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py:138: UserWarning: training_step returned None. If this was on purpose, ignore this warning...
self.warning_cache.warn("training_step returned None. If this was on purpose, ignore this warning...")
Epoch 0: 75%|███████▌ | 12/16 [00:18<00:06, 1.57s/it, loss=nan]
2023-04-26 17:48:24,493 - anomalib.models.padim.lightning_model - INFO - Aggregating the embedding extracted from the training set.
Validation: 0it [00:00, ?it/s]2023-04-26 17:48:24,712 - anomalib.models.padim.lightning_model - INFO - Fitting a Gaussian to the embedding collected from the training set.

Validation: 0%| | 0/4 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/4 [00:00<?, ?it/s]
Epoch 0: 81%|████████▏ | 13/16 [01:07<00:15, 5.22s/it, loss=nan]
Epoch 0: 88%|████████▊ | 14/16 [01:08<00:09, 4.88s/it, loss=nan]
Epoch 0: 94%|█████████▍| 15/16 [01:08<00:04, 4.60s/it, loss=nan]
Epoch 0: 100%|██████████| 16/16 [01:09<00:00, 4.36s/it, loss=nan, image_F1Score=0.480, image_AUROC=0.793]
C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\torchmetrics\utilities\prints.py:36: DeprecationWarning: torchmetrics.functional.auc has been move to torchmetrics.utilities.compute in v0.10 and will be removed in v0.11.
warnings.warn(*args, **kwargs)
Epoch 0: 100%|██████████| 16/16 [01:09<00:00, 4.36s/it, loss=nan, image_F1Score=0.480, image_AUROC=0.793]Traceback (most recent call last):
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\torch\serialization.py", line 379, in save
_save(obj, opened_zipfile, pickle_module, pickle_protocol)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\torch\serialization.py", line 604, in _save
zip_file.write_record(name, storage.data_ptr(), num_bytes)
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\torch\serialization.py", line 380, in save
return
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\torch\serialization.py", line 259, in exit
self.file_like.write_end_of_file()
RuntimeError: [enforce fail at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\caffe2\serialize\inline_container.cc:319] . unexpected pos 98176448 vs 98176336

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:/MyCode/4_HC/Time/202304/Anomalib_Seg/anomalib-main/tools/train.py", line 76, in
train()
File "D:/MyCode/4_HC/Time/202304/Anomalib_Seg/anomalib-main/tools/train.py", line 61, in train
trainer.fit(model=model, datamodule=datamodule)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 608, in fit
call._call_and_handle_interrupt(
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\trainer\call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 650, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1112, in _run
results = self._run_stage()
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1191, in _run_stage
self._run_train()
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1214, in _run_train
self.fit_loop.run()

File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\loops\loop.py", line 200, in run
self.on_advance_end()
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 295, in on_advance_end
self.trainer._call_callback_hooks("on_train_epoch_end")
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1394, in _call_callback_hooks
fn(self, self.lightning_module, *args, **kwargs)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 304, in on_train_epoch_end
self._save_topk_checkpoint(trainer, monitor_candidates)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 363, in _save_topk_checkpoint
self._save_none_monitor_checkpoint(trainer, monitor_candidates)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 669, in _save_none_monitor_checkpoint
self._save_checkpoint(trainer, filepath)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 366, in _save_checkpoint
trainer.save_checkpoint(filepath, self.save_weights_only)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1939, in save_checkpoint
self._checkpoint_connector.save_checkpoint(filepath, weights_only=weights_only, storage_options=storage_options)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\trainer\connectors\checkpoint_connector.py", line 511, in save_checkpoint
self.trainer.strategy.save_checkpoint(_checkpoint, filepath, storage_options=storage_options)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 466, in save_checkpoint
self.checkpoint_io.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\lightning_fabric\plugins\io\torch_io.py", line 54, in save_checkpoint
_atomic_save(checkpoint, path)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\lightning_fabric\utilities\cloud_io.py", line 67, in _atomic_save
torch.save(checkpoint, bytesbuffer)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\torch\serialization.py", line 381, in save
_legacy_save(obj, opened_file, pickle_module, pickle_protocol)
File "C:\ProgramData\anaconda3\envs\HC_Anomalib\lib\site-packages\torch\serialization.py", line 225, in exit
self.file_like.flush()
ValueError: I/O operation on closed file.

进程已结束,退出代码-1073740791 (0xC0000409)

Pip/GitHub

pip

What version/branch did you use?

latest

Configuration YAML

Same as the official Yaml, only the dataset part has been replaced with its own dataset

Logs

None

Code of Conduct

• I agree to follow this project's Code of Conduct

[samet-akcay]

Thanks for your patience @laogonggong847. I'll try to reply to your questions one by one.

Padim Training with WideResNet-50

I'm unfortunately unable to reproduce this. Are you certain that you do not change your configuration file? When I use the default settings by just changing the backbone to wide_resnet50_2, I can train the model. Can you share your config file. For the record, here is my output after the training:

python tools/train.py --model padim
/home/sakcay/Projects/anomalib/src/anomalib/config/config.py:280: UserWarning: config.project.unique_dir is set to False. This does not ensure that your results will be written in an empty directory and you may overwrite files.
  warn(
Global seed set to 42
2023-08-11 07:15:33,911 - anomalib.data - INFO - Loading the datamodule
2023-08-11 07:15:33,911 - anomalib.data.utils.transform - INFO - No config file has been provided. Using default transforms.
2023-08-11 07:15:33,911 - anomalib.data.utils.transform - INFO - No config file has been provided. Using default transforms.
2023-08-11 07:15:33,912 - anomalib.models - INFO - Loading the model.
2023-08-11 07:15:33,912 - anomalib.models.components.base.anomaly_module - INFO - Initializing PadimLightning model.
/home/sakcay/.local/lib/python3.10/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric `PrecisionRecallCurve` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
  warnings.warn(*args, **kwargs)
2023-08-11 07:15:33,916 - anomalib.models.components.feature_extractors.timm - WARNING - FeatureExtractor is deprecated. Use TimmFeatureExtractor instead. Both FeatureExtractor and TimmFeatureExtractor will be removed in a future release.
2023-08-11 07:15:35,087 - timm.models.helpers - INFO - Loading pretrained weights from url (https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/wide_resnet50_racm-8234f177.pth)
2023-08-11 07:15:35,549 - anomalib.utils.loggers - INFO - Loading the experiment logger(s)
2023-08-11 07:15:35,550 - anomalib.utils.callbacks - INFO - Loading the callbacks
/home/sakcay/Projects/anomalib/src/anomalib/utils/callbacks/__init__.py:142: UserWarning: Export option: None not found. Defaulting to no model export
  warnings.warn(f"Export option: {config.optimization.export_mode} not found. Defaulting to no model export")
2023-08-11 07:15:35,561 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True
2023-08-11 07:15:35,561 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores
2023-08-11 07:15:35,561 - pytorch_lightning.utilities.rank_zero - INFO - IPU available: False, using: 0 IPUs
2023-08-11 07:15:35,561 - pytorch_lightning.utilities.rank_zero - INFO - HPU available: False, using: 0 HPUs
2023-08-11 07:15:35,561 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
2023-08-11 07:15:35,561 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..
2023-08-11 07:15:35,561 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..
2023-08-11 07:15:35,561 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer(limit_predict_batches=1.0)` was configured so 100% of the batches will be used..
2023-08-11 07:15:35,561 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
2023-08-11 07:15:35,561 - anomalib - INFO - Training the model.
2023-08-11 07:15:35,664 - anomalib.data.mvtec - INFO - Found the dataset.
2023-08-11 07:15:35,685 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
/home/sakcay/.local/lib/python3.10/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric `ROC` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
  warnings.warn(*args, **kwargs)
/home/sakcay/.local/lib/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:613: UserWarning: Checkpoint directory /home/sakcay/Projects/anomalib/results/padim/mvtec/bottle/run/weights/lightning exists and is not empty.
  rank_zero_warn(f"Checkpoint directory {dirpath} exists and is not empty.")
2023-08-11 07:15:36,684 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
/home/sakcay/.local/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py:183: UserWarning: `LightningModule.configure_optimizers` returned `None`, this fit will run with no optimizer
  rank_zero_warn(
2023-08-11 07:15:36,688 - pytorch_lightning.callbacks.model_summary - INFO - 
  | Name                  | Type                     | Params
-------------------------------------------------------------------
0 | image_threshold       | AnomalyScoreThreshold    | 0     
1 | pixel_threshold       | AnomalyScoreThreshold    | 0     
2 | model                 | PadimModel               | 24.9 M
3 | image_metrics         | AnomalibMetricCollection | 0     
4 | pixel_metrics         | AnomalibMetricCollection | 0     
5 | normalization_metrics | MinMax                   | 0     
-------------------------------------------------------------------
24.9 M    Trainable params
0         Non-trainable params
24.9 M    Total params
99.450    Total estimated model params size (MB)
Epoch 0:   0%|                                                                                                                        | 0/10 [00:00<?, ?it/s]/home/sakcay/.local/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py:138: UserWarning: `training_step` returned `None`. If this was on purpose, ignore this warning...
  self.warning_cache.warn("`training_step` returned `None`. If this was on purpose, ignore this warning...")
Epoch 0:  70%|████████████████2023-08-11 07:15:39,024 - anomalib.models.padim.lightning_model - INFO - Aggregating the embedding extracted from the training set.dation: 0it [00:00, ?it/s]
2023-08-11 07:15:39,176 - anomalib.models.padim.lightning_model - INFO - Fitting a Gaussian to the embedding collected from the training set.
Epoch 0: 100%|█████████████████████████████████████████████████████████████| 10/10 [00:31<00:00,  3.13s/it, loss=nan, pixel_F1Score=0.705, pixel_AUROC=0.983]2023-08-11 07:16:12,716 - pytorch_lightning.utilities.rank_zero - INFO - `Trainer.fit` stopped: `max_epochs=1` reached.                                      
Epoch 0: 100%|█████████████████████████████████████████████████████████████| 10/10 [00:36<00:00,  3.60s/it, loss=nan, pixel_F1Score=0.705, pixel_AUROC=0.983]
2023-08-11 07:16:12,846 - anomalib.utils.callbacks.timer - INFO - Training took 36.16 seconds
2023-08-11 07:16:12,846 - anomalib - INFO - Loading the best model weights.
2023-08-11 07:16:12,846 - anomalib - INFO - Testing the model.
2023-08-11 07:16:12,850 - anomalib.data.mvtec - INFO - Found the dataset.
2023-08-11 07:16:12,850 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
2023-08-11 07:16:12,851 - anomalib.utils.callbacks.model_loader - INFO - Loading the model from /home/sakcay/Projects/anomalib/results/padim/mvtec/bottle/run/weights/lightning/model-v2.ckpt
2023-08-11 07:16:14,951 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
Testing DataLoader 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:14<00:00,  4.85s/it]2023-08-11 07:16:30,522 - anomalib.utils.callbacks.timer - INFO - Testing took 15.568396091461182 seconds
Throughput (batch_size=32) : 5.331313483572217 FPS
Testing DataLoader 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:14<00:00,  4.91s/it]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        image_AUROC        │    0.9976190328598022     │
│       image_F1Score       │    0.9919999837875366     │
│        pixel_AUROC        │    0.9825913310050964     │
│       pixel_F1Score       │    0.7047178745269775     │
└───────────────────────────┴───────────────────────────┘

I feel the issue could be related to something else.

Patchcore ONNX Runtime

This has been fixed in one of the recent anomalib versions, so ONNX version of the patchcore model should ideally work fine.

How to Calculate the Confidence Score and Label

Anomalib inferencers have post_process methods that takes the predictions of the model and apply post-processing to the raw predictions. For example, the following code base show how OpenVINO inference applies post-processing using the meta data, which stores the relevant information such as image_threshold and pixel_threshold.
https://github.com/openvinotoolkit/anomalib/blob/f2c5a013bc6f3fd773ad083ee11102817322dd76/src/anomalib/deploy/inferencers/openvino_inferencer.py#L128-L201

Please note that you could use OpenVINO Inferencer to run the ONNX model. You don't need to write a separate script for ONNX.

Let us know if you have any other questions.

[laogonggong847]

Yes thank you very much, I tested it and everything is working fine, thank you very much!