Is OpenVINO really accelerates Smart Search job?

[DX37]

I've been testing OpenVINO a while. Not sure that it accelerating anything. On the contrary, it uses more threads in gunicorn and also uses render from GPU, while providing same results in seconds per jobs.

Here's what I've got with Smart Search job on Intel UHD Graphics 630 in i5-10500.

OpenVINO, 2 concurrent jobs, ~2 assets per 5 seconds

OpenVINO, 4 concurrent jobs, ~4 assets per 9 seconds

And here's results with CPU, but with the same time.

CPU, 2 concurrent jobs, ~2 assets per 5 seconds

CPU, 4 concurrent jobs, ~4 assets per 9 seconds

So, is it me doing something wrong, or is it OpenVINO that is supposed to act like this, or is it something else?

[DX37]

Or is it model dependent? I'm using nllb-clip-large-siglip__v1

[mertalev]

It is, to some extent. Certain models can get optimized better than others, etc. But I doubt that's the cause here.

[DX37]

I wonder if Immich is still sending assets to CPU provider.

[mertalev]

No, ONNX Runtime decides where parts of the model are allocated. It will put almost everything on the OpenVINO provider except for specific things that are faster to do on CPU.

[mertalev]

There are too many factors to say anything decisive, but some thoughts:

• Models are compiled to OpenVINO format on the first load. This can take a few minutes, especially with a large model.
• The GPU utilization looks very spotty. It might not be getting enough work to do.
• Relatedly, inputs to the GPU aren't batched right now. OpenVINO may be more sensitive to this than CUDA.
• The GPU isn't very strong. It might not be powerful enough to make a big difference.