[Bug] Training us "CFA" on category "carpet" and "tile" of MvTec lead to error

[cugwu]

Hi I'm using CFA with wide_resnet50_2 on MvTeC. When CFA is trained on the category "carpet" and "tile" the training doesn't complete and I get the following error for both datasets:

Traceback (most recent call last):
  File "../tools/train.py", line 79, in <module>
    train(args)
  File "../tools/train.py", line 64, in train
    trainer.fit(model=model, datamodule=datamodule)
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
    results = self._run_stage()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage
    self._run_train()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_train
    self.fit_loop.run()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.on_advance_end()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 250, in on_advance_end
    self._run_validation()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 308, in _run_validation
    self.val_loop.run()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 206, in run
    output = self.on_run_end()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 184, in on_run_end
    self._on_evaluation_epoch_end()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 296, in _on_evaluation_epoch_end
    self.trainer._logger_connector.on_epoch_end()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 192, in on_epoch_end
    metrics = self.metrics
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 237, in metrics
    return self.trainer._results.metrics(on_step)
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 586, in metrics
    value = apply_to_collection(result_metric, _ResultMetric, self._get_cache, on_step, include_none=False)
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/lightning_utilities/core/apply_func.py", line 51, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 545, in _get_cache
    result_metric.compute()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 300, in wrapped_func
    self._computed = compute(*args, **kwargs)
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 269, in compute
    return self.value.compute()
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/metric.py", line 534, in wrapped_func
    value = compute(*args, **kwargs)
  File "/home/clusterusers/cugwu/codes/cvl/F4AD/ad_models/anomalib/src/anomalib/utils/metrics/aupro.py", line 217, in compute
    fpr, tpr = self._compute()
  File "/home/clusterusers/cugwu/codes/cvl/F4AD/ad_models/anomalib/src/anomalib/utils/metrics/aupro.py", line 205, in _compute
    cca = self.perform_cca().flatten()
  File "/home/clusterusers/cugwu/codes/cvl/F4AD/ad_models/anomalib/src/anomalib/utils/metrics/aupro.py", line 84, in perform_cca
    target = dim_zero_cat(self.target)
  File "/data/users/cugwu/all_envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/utilities/data.py", line 47, in dim_zero_cat
    return torch.cat(x, dim=0)
RuntimeError: Tensors must have same number of dimensions: got 3 and 2

The logs, as you can see below, stop at the validation phase:

Training: 0it [00:00, ?it/s]
Training:   0%|          | 0/88 [00:00<?, ?it/s]
Epoch 0:   0%|          | 0/88 [00:00<?, ?it/s] 
Epoch 0:   1%|          | 1/88 [00:01<02:00,  1.39s/it]
Epoch 0:   1%|          | 1/88 [00:01<02:00,  1.39s/it, loss=1.79e+03]
Epoch 0:   2%|▏         | 2/88 [00:01<01:02,  1.38it/s, loss=1.79e+03]
Epoch 0:   2%|▏         | 2/88 [00:01<01:03,  1.35it/s, loss=1.57e+04]
Epoch 0:   3%|▎         | 3/88 [00:01<00:43,  1.96it/s, loss=1.57e+04]
Epoch 0:   3%|▎         | 3/88 [00:01<00:44,  1.92it/s, loss=1.19e+04]
Epoch 0:   5%|▍         | 4/88 [00:01<00:34,  2.47it/s, loss=1.19e+04]
Epoch 0:   5%|▍         | 4/88 [00:01<00:34,  2.42it/s, loss=1.12e+04]
Epoch 0:   6%|▌         | 5/88 [00:01<00:28,  2.87it/s, loss=1.12e+04]
Epoch 0:   6%|▌         | 5/88 [00:01<00:29,  2.82it/s, loss=1.25e+04]
Epoch 0:   7%|▋         | 6/88 [00:01<00:25,  3.22it/s, loss=1.25e+04]
Epoch 0:   7%|▋         | 6/88 [00:01<00:25,  3.17it/s, loss=1.27e+04]
Epoch 0:   8%|▊         | 7/88 [00:01<00:23,  3.52it/s, loss=1.27e+04]
Epoch 0:   8%|▊         | 7/88 [00:02<00:23,  3.47it/s, loss=1.15e+04]
Epoch 0:   9%|▉         | 8/88 [00:02<00:21,  3.79it/s, loss=1.15e+04]
Epoch 0:   9%|▉         | 8/88 [00:02<00:21,  3.74it/s, loss=1.05e+04]
Epoch 0:  10%|█         | 9/88 [00:02<00:19,  4.00it/s, loss=1.05e+04]
Epoch 0:  10%|█         | 9/88 [00:02<00:19,  3.95it/s, loss=1.01e+04]
Epoch 0:  11%|█▏        | 10/88 [00:02<00:18,  4.23it/s, loss=1.01e+04]
Epoch 0:  11%|█▏        | 10/88 [00:02<00:18,  4.18it/s, loss=9.94e+03]
Epoch 0:  12%|█▎        | 11/88 [00:02<00:17,  4.41it/s, loss=9.94e+03]
Epoch 0:  12%|█▎        | 11/88 [00:02<00:17,  4.37it/s, loss=9.72e+03]
Epoch 0:  14%|█▎        | 12/88 [00:02<00:16,  4.60it/s, loss=9.72e+03]
Epoch 0:  14%|█▎        | 12/88 [00:02<00:16,  4.55it/s, loss=9.26e+03]
Epoch 0:  15%|█▍        | 13/88 [00:02<00:15,  4.76it/s, loss=9.26e+03]
Epoch 0:  15%|█▍        | 13/88 [00:02<00:15,  4.71it/s, loss=8.76e+03]
Epoch 0:  16%|█▌        | 14/88 [00:02<00:15,  4.90it/s, loss=8.76e+03]
Epoch 0:  16%|█▌        | 14/88 [00:02<00:15,  4.84it/s, loss=8.33e+03]
Epoch 0:  17%|█▋        | 15/88 [00:02<00:14,  5.01it/s, loss=8.33e+03]
Epoch 0:  17%|█▋        | 15/88 [00:03<00:14,  4.96it/s, loss=8.09e+03]
Epoch 0:  18%|█▊        | 16/88 [00:03<00:13,  5.15it/s, loss=8.09e+03]
Epoch 0:  18%|█▊        | 16/88 [00:03<00:14,  5.10it/s, loss=7.91e+03]
Epoch 0:  19%|█▉        | 17/88 [00:03<00:13,  5.23it/s, loss=7.91e+03]
Epoch 0:  19%|█▉        | 17/88 [00:03<00:13,  5.19it/s, loss=7.69e+03]
Epoch 0:  20%|██        | 18/88 [00:03<00:13,  5.33it/s, loss=7.69e+03]
Epoch 0:  20%|██        | 18/88 [00:03<00:13,  5.28it/s, loss=7.43e+03]
Epoch 0:  22%|██▏       | 19/88 [00:03<00:12,  5.44it/s, loss=7.43e+03]
Epoch 0:  22%|██▏       | 19/88 [00:03<00:12,  5.40it/s, loss=7.16e+03]
Epoch 0:  23%|██▎       | 20/88 [00:03<00:12,  5.53it/s, loss=7.16e+03]
Epoch 0:  23%|██▎       | 20/88 [00:03<00:12,  5.49it/s, loss=6.92e+03]
Epoch 0:  24%|██▍       | 21/88 [00:03<00:11,  5.62it/s, loss=6.92e+03]
Epoch 0:  24%|██▍       | 21/88 [00:03<00:12,  5.58it/s, loss=6.99e+03]
Epoch 0:  25%|██▌       | 22/88 [00:03<00:11,  5.70it/s, loss=6.99e+03]
Epoch 0:  25%|██▌       | 22/88 [00:03<00:11,  5.65it/s, loss=5.65e+03]
Epoch 0:  26%|██▌       | 23/88 [00:03<00:11,  5.77it/s, loss=5.65e+03]
Epoch 0:  26%|██▌       | 23/88 [00:04<00:11,  5.73it/s, loss=5.58e+03]
Epoch 0:  27%|██▋       | 24/88 [00:04<00:10,  5.84it/s, loss=5.58e+03]
Epoch 0:  27%|██▋       | 24/88 [00:04<00:11,  5.80it/s, loss=5.23e+03]
Epoch 0:  28%|██▊       | 25/88 [00:04<00:10,  5.91it/s, loss=5.23e+03]
Epoch 0:  28%|██▊       | 25/88 [00:04<00:10,  5.86it/s, loss=4.45e+03]
Epoch 0:  30%|██▉       | 26/88 [00:04<00:10,  5.96it/s, loss=4.45e+03]
Epoch 0:  30%|██▉       | 26/88 [00:04<00:10,  5.92it/s, loss=3.86e+03]
Epoch 0:  31%|███       | 27/88 [00:04<00:10,  6.01it/s, loss=3.86e+03]
Epoch 0:  31%|███       | 27/88 [00:04<00:10,  5.98it/s, loss=3.75e+03]
Epoch 0:  32%|███▏      | 28/88 [00:04<00:09,  6.07it/s, loss=3.75e+03]
Epoch 0:  32%|███▏      | 28/88 [00:04<00:09,  6.04it/s, loss=3.7e+03] 
Epoch 0:  33%|███▎      | 29/88 [00:04<00:09,  6.13it/s, loss=3.7e+03]
Epoch 0:  33%|███▎      | 29/88 [00:04<00:09,  6.09it/s, loss=3.47e+03]
Epoch 0:  34%|███▍      | 30/88 [00:04<00:09,  6.18it/s, loss=3.47e+03]
Epoch 0:  34%|███▍      | 30/88 [00:04<00:09,  6.13it/s, loss=3.12e+03]
Epoch 0:  35%|███▌      | 31/88 [00:04<00:09,  6.22it/s, loss=3.12e+03]
Epoch 0:  35%|███▌      | 31/88 [00:05<00:09,  6.19it/s, loss=2.82e+03]
Epoch 0:  36%|███▋      | 32/88 [00:05<00:08,  6.25it/s, loss=2.82e+03]
Epoch 0:  36%|███▋      | 32/88 [00:05<00:09,  6.22it/s, loss=2.7e+03] 
Epoch 0:  38%|███▊      | 33/88 [00:05<00:08,  6.31it/s, loss=2.7e+03]
Epoch 0:  38%|███▊      | 33/88 [00:05<00:08,  6.28it/s, loss=2.64e+03]
Epoch 0:  39%|███▊      | 34/88 [00:05<00:08,  6.36it/s, loss=2.64e+03]
Epoch 0:  39%|███▊      | 34/88 [00:05<00:08,  6.32it/s, loss=2.6e+03] 
Epoch 0:  40%|███▉      | 35/88 [00:05<00:08,  6.39it/s, loss=2.6e+03]
Epoch 0:  40%|███▉      | 35/88 [00:05<00:08,  6.37it/s, loss=2.45e+03]
Epoch 0:  41%|████      | 36/88 [00:05<00:08,  6.43it/s, loss=2.45e+03]
Epoch 0:  41%|████      | 36/88 [00:05<00:08,  6.40it/s, loss=2.26e+03]
Epoch 0:  42%|████▏     | 37/88 [00:05<00:07,  6.47it/s, loss=2.26e+03]
Epoch 0:  42%|████▏     | 37/88 [00:05<00:07,  6.45it/s, loss=2.14e+03]
Epoch 0:  43%|████▎     | 38/88 [00:05<00:07,  6.51it/s, loss=2.14e+03]
Epoch 0:  43%|████▎     | 38/88 [00:05<00:07,  6.48it/s, loss=2.07e+03]
Epoch 0:  44%|████▍     | 39/88 [00:05<00:07,  6.54it/s, loss=2.07e+03]
Epoch 0:  44%|████▍     | 39/88 [00:05<00:07,  6.51it/s, loss=2.03e+03]
Epoch 0:  45%|████▌     | 40/88 [00:06<00:07,  6.56it/s, loss=2.03e+03]
Epoch 0:  45%|████▌     | 40/88 [00:06<00:07,  6.53it/s, loss=1.99e+03]
Epoch 0:  47%|████▋     | 41/88 [00:06<00:07,  6.59it/s, loss=1.99e+03]
Epoch 0:  47%|████▋     | 41/88 [00:06<00:07,  6.56it/s, loss=1.9e+03] 
Epoch 0:  48%|████▊     | 42/88 [00:06<00:06,  6.62it/s, loss=1.9e+03]
Epoch 0:  48%|████▊     | 42/88 [00:06<00:06,  6.59it/s, loss=1.82e+03]
Epoch 0:  49%|████▉     | 43/88 [00:06<00:06,  6.65it/s, loss=1.82e+03]
Epoch 0:  49%|████▉     | 43/88 [00:06<00:06,  6.62it/s, loss=1.74e+03]
Epoch 0:  50%|█████     | 44/88 [00:06<00:06,  6.66it/s, loss=1.74e+03]
Epoch 0:  50%|█████     | 44/88 [00:06<00:06,  6.63it/s, loss=1.69e+03]
Epoch 0:  51%|█████     | 45/88 [00:06<00:06,  6.70it/s, loss=1.69e+03]
Epoch 0:  51%|█████     | 45/88 [00:06<00:06,  6.66it/s, loss=1.66e+03]
Epoch 0:  52%|█████▏    | 46/88 [00:06<00:06,  6.72it/s, loss=1.66e+03]
Epoch 0:  52%|█████▏    | 46/88 [00:06<00:06,  6.69it/s, loss=1.62e+03]
Epoch 0:  53%|█████▎    | 47/88 [00:06<00:06,  6.74it/s, loss=1.62e+03]
Epoch 0:  53%|█████▎    | 47/88 [00:07<00:06,  6.71it/s, loss=1.57e+03]
Epoch 0:  55%|█████▍    | 48/88 [00:07<00:05,  6.76it/s, loss=1.57e+03]
Epoch 0:  55%|█████▍    | 48/88 [00:07<00:05,  6.73it/s, loss=1.51e+03]
Epoch 0:  56%|█████▌    | 49/88 [00:07<00:05,  6.79it/s, loss=1.51e+03]
Epoch 0:  56%|█████▌    | 49/88 [00:07<00:05,  6.76it/s, loss=1.45e+03]
Epoch 0:  57%|█████▋    | 50/88 [00:07<00:05,  6.81it/s, loss=1.45e+03]
Epoch 0:  57%|█████▋    | 50/88 [00:07<00:05,  6.78it/s, loss=1.42e+03]
Epoch 0:  58%|█████▊    | 51/88 [00:07<00:05,  6.83it/s, loss=1.42e+03]
Epoch 0:  58%|█████▊    | 51/88 [00:07<00:05,  6.81it/s, loss=1.39e+03]
Epoch 0:  59%|█████▉    | 52/88 [00:07<00:05,  6.84it/s, loss=1.39e+03]
Epoch 0:  59%|█████▉    | 52/88 [00:07<00:05,  6.81it/s, loss=1.36e+03]
Epoch 0:  60%|██████    | 53/88 [00:07<00:05,  6.85it/s, loss=1.36e+03]
Epoch 0:  60%|██████    | 53/88 [00:07<00:05,  6.83it/s, loss=1.32e+03]
Epoch 0:  61%|██████▏   | 54/88 [00:07<00:04,  6.88it/s, loss=1.32e+03]
Epoch 0:  61%|██████▏   | 54/88 [00:07<00:04,  6.85it/s, loss=1.27e+03]
Epoch 0:  62%|██████▎   | 55/88 [00:07<00:04,  6.89it/s, loss=1.27e+03]
Epoch 0:  62%|██████▎   | 55/88 [00:08<00:04,  6.87it/s, loss=1.24e+03]
Epoch 0:  64%|██████▎   | 56/88 [00:08<00:04,  6.91it/s, loss=1.24e+03]
Epoch 0:  64%|██████▎   | 56/88 [00:08<00:04,  6.88it/s, loss=1.22e+03]
Epoch 0:  65%|██████▍   | 57/88 [00:08<00:04,  6.93it/s, loss=1.22e+03]
Epoch 0:  65%|██████▍   | 57/88 [00:08<00:04,  6.91it/s, loss=1.18e+03]
Epoch 0:  66%|██████▌   | 58/88 [00:08<00:04,  6.99it/s, loss=1.18e+03]
Epoch 0:  66%|██████▌   | 58/88 [00:08<00:04,  6.99it/s, loss=1.15e+03]

Validation: 0it [00:00, ?it/s]�[A

Validation:   0%|          | 0/30 [00:00<?, ?it/s]�[A

Validation DataLoader 0:   0%|          | 0/30 [00:00<?, ?it/s]�[A

Validation DataLoader 0:   3%|▎         | 1/30 [00:00<00:01, 18.21it/s]�[A
Epoch 0:  67%|██████▋   | 59/88 [00:09<00:04,  6.30it/s, loss=1.15e+03]

Validation DataLoader 0:   7%|▋         | 2/30 [00:00<00:02, 13.40it/s]�[A
Epoch 0:  68%|██████▊   | 60/88 [00:09<00:04,  6.34it/s, loss=1.15e+03]

Validation DataLoader 0:  10%|█         | 3/30 [00:00<00:02, 11.49it/s]�[A
Epoch 0:  69%|██████▉   | 61/88 [00:09<00:04,  6.37it/s, loss=1.15e+03]

Validation DataLoader 0:  13%|█▎        | 4/30 [00:00<00:02,  9.99it/s]�[A
Epoch 0:  70%|███████   | 62/88 [00:09<00:04,  6.38it/s, loss=1.15e+03]

Validation DataLoader 0:  17%|█▋        | 5/30 [00:00<00:02,  9.98it/s]�[A
Epoch 0:  72%|███████▏  | 63/88 [00:09<00:03,  6.42it/s, loss=1.15e+03]

Validation DataLoader 0:  20%|██        | 6/30 [00:00<00:02,  9.69it/s]�[A
Epoch 0:  73%|███████▎  | 64/88 [00:09<00:03,  6.45it/s, loss=1.15e+03]

Validation DataLoader 0:  23%|██▎       | 7/30 [00:00<00:02,  9.44it/s]�[A
Epoch 0:  74%|███████▍  | 65/88 [00:10<00:03,  6.47it/s, loss=1.15e+03]

Validation DataLoader 0:  27%|██▋       | 8/30 [00:00<00:02,  9.29it/s]�[A
Epoch 0:  75%|███████▌  | 66/88 [00:10<00:03,  6.49it/s, loss=1.15e+03]

Validation DataLoader 0:  30%|███       | 9/30 [00:00<00:02,  9.17it/s]�[A
Epoch 0:  76%|███████▌  | 67/88 [00:10<00:03,  6.51it/s, loss=1.15e+03]

Validation DataLoader 0:  33%|███▎      | 10/30 [00:01<00:02,  9.08it/s]�[A
Epoch 0:  77%|███████▋  | 68/88 [00:10<00:03,  6.53it/s, loss=1.15e+03]

Validation DataLoader 0:  37%|███▋      | 11/30 [00:01<00:02,  9.15it/s]�[A
Epoch 0:  78%|███████▊  | 69/88 [00:10<00:02,  6.56it/s, loss=1.15e+03]

Validation DataLoader 0:  40%|████      | 12/30 [00:01<00:01,  9.15it/s]�[A
Epoch 0:  80%|███████▉  | 70/88 [00:10<00:02,  6.59it/s, loss=1.15e+03]

Validation DataLoader 0:  43%|████▎     | 13/30 [00:01<00:01,  9.12it/s]�[A
Epoch 0:  81%|████████  | 71/88 [00:10<00:02,  6.61it/s, loss=1.15e+03]

Validation DataLoader 0:  47%|████▋     | 14/30 [00:01<00:01,  9.11it/s]�[A
Epoch 0:  82%|████████▏ | 72/88 [00:10<00:02,  6.64it/s, loss=1.15e+03]

Validation DataLoader 0:  50%|█████     | 15/30 [00:01<00:01,  9.10it/s]�[A
Epoch 0:  83%|████████▎ | 73/88 [00:10<00:02,  6.66it/s, loss=1.15e+03]

Validation DataLoader 0:  53%|█████▎    | 16/30 [00:01<00:01,  9.10it/s]�[A
Epoch 0:  84%|████████▍ | 74/88 [00:11<00:02,  6.69it/s, loss=1.15e+03]

Validation DataLoader 0:  57%|█████▋    | 17/30 [00:01<00:01,  9.08it/s]�[A
Epoch 0:  85%|████████▌ | 75/88 [00:11<00:01,  6.71it/s, loss=1.15e+03]

Validation DataLoader 0:  60%|██████    | 18/30 [00:01<00:01,  9.04it/s]�[A
Epoch 0:  86%|████████▋ | 76/88 [00:11<00:01,  6.72it/s, loss=1.15e+03]

Validation DataLoader 0:  63%|██████▎   | 19/30 [00:02<00:01,  9.01it/s]�[A
Epoch 0:  88%|████████▊ | 77/88 [00:11<00:01,  6.74it/s, loss=1.15e+03]

Validation DataLoader 0:  67%|██████▋   | 20/30 [00:02<00:01,  8.97it/s]�[A
Epoch 0:  89%|████████▊ | 78/88 [00:11<00:01,  6.76it/s, loss=1.15e+03]

Validation DataLoader 0:  70%|███████   | 21/30 [00:02<00:01,  8.92it/s]�[A
Epoch 0:  90%|████████▉ | 79/88 [00:11<00:01,  6.77it/s, loss=1.15e+03]

Validation DataLoader 0:  73%|███████▎  | 22/30 [00:02<00:00,  8.87it/s]�[A
Epoch 0:  91%|█████████ | 80/88 [00:11<00:01,  6.78it/s, loss=1.15e+03]

Validation DataLoader 0:  77%|███████▋  | 23/30 [00:02<00:00,  8.87it/s]�[A
Epoch 0:  92%|█████████▏| 81/88 [00:11<00:01,  6.80it/s, loss=1.15e+03]

Validation DataLoader 0:  80%|████████  | 24/30 [00:02<00:00,  8.85it/s]�[A
Epoch 0:  93%|█████████▎| 82/88 [00:12<00:00,  6.82it/s, loss=1.15e+03]

Validation DataLoader 0:  83%|████████▎ | 25/30 [00:02<00:00,  8.75it/s]�[A
Epoch 0:  94%|█████████▍| 83/88 [00:12<00:00,  6.82it/s, loss=1.15e+03]

Validation DataLoader 0:  87%|████████▋ | 26/30 [00:02<00:00,  8.79it/s]�[A
Epoch 0:  95%|█████████▌| 84/88 [00:12<00:00,  6.85it/s, loss=1.15e+03]

Validation DataLoader 0:  90%|█████████ | 27/30 [00:03<00:00,  8.74it/s]�[A
Epoch 0:  97%|█████████▋| 85/88 [00:12<00:00,  6.86it/s, loss=1.15e+03]

Validation DataLoader 0:  93%|█████████▎| 28/30 [00:03<00:00,  8.76it/s]�[A
Epoch 0:  98%|█████████▊| 86/88 [00:12<00:00,  6.88it/s, loss=1.15e+03]

Validation DataLoader 0:  97%|█████████▋| 29/30 [00:03<00:00,  8.74it/s]�[A
Epoch 0:  99%|█████████▉| 87/88 [00:12<00:00,  6.89it/s, loss=1.15e+03]

Validation DataLoader 0: 100%|██████████| 30/30 [00:03<00:00,  8.93it/s]�[A
Epoch 0: 100%|██████████| 88/88 [00:12<00:00,  6.95it/s, loss=1.15e+03]
Epoch 0: 100%|██████████| 88/88 [00:14<00:00,  6.18it/s, loss=1.15e+03]


                                                                        �[A

Initially, I thought that maybe the images were corrupted, but I trained padim, patchcore, cflow and fastflow on these categories without any problem.
Does someone has an idea of what is happening?

[blaz-r]

Hello. The issue seems to stem from AUPRO metric calculation. From the error message, I think it might be related to mask output, since that can be either something like [B, H, W] or [B, 1, H, W].

Can you clarify, is this only an issue of these two categories or all categories? It'd also help if you could please share the config file and which version of anomalib you are using.

[cugwu]

Hi, yes this is a problem only for these two categories when using CFA. With the other models, I don't have this error.
However, since you mentioned AUPRO, I encountered an issue with AUPRO metrics while using FastFlow. You can find the link to the issue here: #1651

Regarding CFA I'm using Anomalib version: 1.0.0.dev0 and this is the config file:

dataset:
  name: mvtec #options: [mvtec, btech, folder]
  format: mvtec
  path: /data/users/cugwu/ad_data/mvtec
  category: tile
  task: segmentation
  train_batch_size: 4
  eval_batch_size: 4
  inference_batch_size: 4
  num_workers: 1
  image_size: 224 # dimensions to which images are resized (mandatory)
  center_crop: null # dimensions to which images are center-cropped after resizing (optional)
  normalization: imagenet # data distribution to which the images will be normalized: [none, imagenet]
  transform_config:
    train: null
    eval: null
  test_split_mode: from_dir # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)

model:
  name: cfa
  backbone: wide_resnet50_2
  gamma_c: 1
  gamma_d: 1
  num_nearest_neighbors: 3
  num_hard_negative_features: 3
  radius: 1e-5
  lr: 1e-3
  weight_decay: 5e-4
  amsgrad: true
  early_stopping:
    patience: 5
    metric: pixel_AUROC
    mode: max
  normalization_method: min_max # options: [null, min_max, cdf]

metrics:
  image:
    - AUROC
  pixel:
    - AUROC
    - AUPRO
  threshold:
    adaptive: true
    image_default: null
    pixel_default: null

visualization:
  show_images: False # show images on the screen
  save_images: True # save images to the file system
  log_images: True # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 0
  path: /data/users/cugwu/ad_data/results/F4AD/anomalib/IN

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: null # options: torch, onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: 1
  enable_progress_bar: true
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1 # Don't validate before extracting features.
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 30
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0 # Don't validate before extracting features.
  log_every_n_steps: 50
  accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  strategy: null
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  auto_lr_find: false
  replace_sampler_ddp: true
  detect_anomaly: false
  auto_scale_batch_size: false
  plugins: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle

[blaz-r]

I've seen the issue yes, and I'm not sure what seems to be the problem. It's rather unusual that it only happens with these two categories. We'll need to look into it more.

[blaz-r]

@cugwu in case you haven't solved this issue yet: the problem is in the way batches are formed. The last element in batch is just a single item, so it gets dimensions of 2 (h, w) while the first bathces have 3 dims (b, h, w).

This causes problems, so to fix this you need to change the batch size.

This should probably be fixed inside AUPRO I believe, to enable arbitrary bath sizes.