OpenVINOInferencer on GPU

[kareldb]

I've trained a model (Padim with resnet 50_2) with the anomalib package (python). The recall & precision are quite good, so I want to implement it for a demo.

Currently I'm using this code:

inferencer = OpenVINOInferencer(
    path=openvino_model_path,
    metadata=metadata_path,
    device="CPU",
)

predictions = inferencer.predict(image=image)

This works perfectly but only gets around 40 fps. When I try to use it with "gpu", the build fails (as I have Nvidia).

My application requires 1000 - 2000 fps. (which I've been able to do with CNN's).

How can I do interference on the GPU?

Thanks!

My training process outputs multiple model filel types:

- model.bin
- model.onnx
- model.xml
- metadata.json

[samet-akcay]

You could use TorchInferencer with a gpu device option. That'd do a gpu inference on an nvidia gpu.

Alternatively you could also use new Engine API, more specifically Engine.predict method.

OpenVINO is optimized for Intel hardware

[kareldb]

Thanks! Are there code examples for this, since I'm new to the anomalib and find it quite difficult to find example code.

[blaz-r]

As Samet already pointed out, you could use the TorchInferencer, there are some examples in the docs.

However, 1000 FPS is really really high requirement. Even the fastest unsupervised model available inside Anomalib (and I think even in general) EfficientAD reaches around 500 FPS on a RTX A6000. There are some supervised methods, that are probably even faster, but probably not near the 1000 mark.

So if you need speed, go with EfficientAD, but I'm not sure if that can get to 1000FPS without using some optimized format like maybe OpenVINO on a ARC gpu, or some tensort things, but I can't be sure for that as I'm not experienced in that regard.

[haimat]

@blaz-r Hi, I am interested in this as well.
How did you achieve 500 FPS with EfficientAd, for what image dimensions?