Slower inference/testing with EfficientAD than promised.

[Palettenbrett]

Hey,

i read the paper from EfficientAD and was blasted by the speed of this networks inference.
The Authors said it achived very high accruacy with only a few milliseconds of inference.
I used their source code (found here: https://github.com/nelson1425/EfficientAD)
to benchmark their model on my laptop (RTX 3050 4gb) and got a average inference time of 25 ms.
After that i wanted to implement the model using Anomalib. While training, the final accruacy was quite high, but the testing time after training was 555ms or 1.8 FPS.
Am i missing something or is EfficientAD slow when testing on the validation dataset while training?
Should i use inference.py when i want to find the real inference speed?
Help would be much appreciated.

[samet-akcay]

Hi @Palettenbrett, just a note that the link you provided is not the official source code. And author of that repo, @nelson1425, is also one of the contributors of EfficientAD implementation of anomalib. The main contributor is @alexriedel1. I'll let them reply to your question in their free time

[Palettenbrett]

Thanks for letting me know. There are two repos linked in the papers-with-code documentation. I got confused wich one it the correct one.

[alexriedel1]

Hi @Palettenbrett, to test the inference speed of EfficientAD in anomalib you should just measure the timing for model inference in eval mode. You could do something like this for example:

from anomalib.models.efficient_ad import EfficientAd
import torch
from tqdm import tqdm
import time

if __name__ == "__main__":
    x = torch.rand(1,3,256,256).cuda()
    model = EfficientAd().model.eval().cuda()

    time_list = []
    num = 1000
    exclude_first = 10
    for i in tqdm(range(num)):
        torch.cuda.synchronize()
        tic = time.time()
        model(x)
        torch.cuda.synchronize()
        time_list.append(time.time()-tic)
    time_list = time_list[exclude_first:]
    print("     + Frame Per Second: {:.2f}".format(1/(sum(time_list)/(num-exclude_first))))

[Palettenbrett]

I have rewritten your code to function for Padim. I only changed the import and this line.
model = Padim(input_size=[256, 256], layers=["layer1", "layer2", "layer3"], backbone="resnet18", n_features=100).model.eval().cuda()
Is n_features the number of safed features you meant earlier?
Is this a correct method for checking the inference time of Padim?

[alexriedel1]

Im not too familiar with Padim but in this implementation it automatically selects the number of features for the supported backbones, so you dont have to pass n_features.
I think Padims time complexity is constant regarding the number of training examples, so yes you basically can test the timing like this. However i dont know if that will throw an error without previous training of the model

[Palettenbrett]

In my case it seems to work fine. Im very suprised because Padim outperforms EfficientAD by a mile. Padim with resnet18 is able to get 237 FPS while EfficientAD_S only gets 51. Is this normal behaviour?

[alexriedel1]

Like I said I'm not 100% sure if Padim gets slower with the number of training examples so you should definately try a trained model too

[Palettenbrett]

Alright. Thanks for the conversation. I will try a trained model in the future.

[vanhaotruong]

I am using the command to train the efficient_ad model using custom data set.

python tools/train.py --model efficient_ad---config custom.yaml

It could create the *.ckpt

However, when i run TorchInferencer

inferencer = TorchInferencer(
    path=r'C:\anomalib\results\efficient_ad\cubes\run\weights\lightning\model.ckpt',  
    device="auto"
)

It shows the error:

File C:\anomalib\src\anomalib\deploy\inferencers\torch_inferencer.py:43, in TorchInferencer.init(self, path, device)
40 self.device = self._get_device(device)
42 # Load the model weights.
---> 43 self.model = self.load_model(path)
44 self.metadata = self._load_metadata(path)
45 self.transform = A.from_dict(self.metadata["transform"])

File C:\anomalib\src\anomalib\deploy\inferencers\torch_inferencer.py:88, in TorchInferencer.load_model(self, path)
78 def load_model(self, path: str | Path) -> nn.Module:
79 """Load the PyTorch model.
80
81 Args:
(...)
85 (AnomalyModule): PyTorch Lightning model.
86 """
---> 88 model = (torch.load(path, map_location=self.device))["model"]
89 model.eval()
90 return model.to(self.device)

KeyError: 'model'

[tominator95]

@vanhaotruong not sure but maybe you can check the discussion #1337. I think this problem is addressed there.