Dynamic Axes with OpenVINO EP

[agrawalsourav98]

I made a PR recently that fixed a bug related to loading external weights when using OpenVINO (see #7389). But recent testing has revealed that the fix works only in certain scenarios. I tested it using the following command to create a session,

session = onnxruntime.InferenceSession('model.onnx', providers=['OpenVINOExecutionProvider'],provider_options=[{'device_type': 'GPU_FP32'}])

The fix works when device_type is specified and GPU_X is specified. But immich currenlty uses this,

session = onnxruntime.InferenceSession('model.onnx', providers=['OpenVINOExecutionProvider'],provider_options=[{'device_id': 'GPU'}])

which doesn't work.

The issue here seems to be that batch_size is only recognized in the first case and not in any other case.

There are 3 fixes for this that should work with every scenario,

1. Replace batch_size with -1. This would mean reverting to something similar that was done earlier (changing inputs and outputs) but would support batches greater than 1.
2. If we are sure, that inference would always be single batch (I am not sure how concurrency works currently), we can get rid of the dynamic axes stuff when exporting models to hugging_face.
3. Instead of using device_id, we use device_type in provider_options and anywhere there is no GPU device available, we just use the CPUExecutionProvider as there would be hardly any performance gain when using OpenVINOExecutionProvider with CPU

IMO the simplest would be 2 if we only do single batch inference or 1 which would retain similar behaviour as other EPs.

Previous discussion at #6869

What do you think @mertalev?

[mertalev]

Thanks for looking more into this!

1. I don't see how this would fix it. Since batch_size is just a name for the dynamic axis, -1 shouldn't behave differently in this case.

2. I intend to add dynamic batching, so making all the models static isn't ideal.

3. This seems like the best solution to me. It's a small change that doesn't affect the exported models.

[agrawalsourav98]

I just ran inference and this is what I had observed. I will verify using openvino library that the dynamic batch is indeed passed as it and works as expected and then try to see if there is anything we can do about this. As, batching is something that we are interested in the future, it makes sense to verify once.

Yeah, 3 seems like an easy solution which works.

[mertalev]

That's interesting if batch_size doesn't work, but -1 does. That's definitely a bug.

Also yeah, it'd be great to test that the dynamic axis actually works 😅

[agrawalsourav98]

Yeah. I did check the openvino support matrix here and I think it would be a waste of time to pursue/debug this (as the support is partial and maybe that's why it does not with with device_id) . Implementing option 3 would mean it works with dynamic batching as well. It should work for integrated gpus just fine but dedicated gpus might be an issue which I cannot debug ATM as I don't have one.

[KoffeinKaio]

If you want to test stuff on a Arc A380 I can do that for you.
preferably in some docker way

[agrawalsourav98]

Yeah, that would be good. I would provide a simple python script if you run that inside the openvino docker image and get the output that would be helpful.

[agrawalsourav98]

@KoffeinKaio Can you run this python script and share the output?
test_openvino.zip

[agrawalsourav98]

Okay. This only shows a single GPU. Is it the external GPU or integrated GPU? Also, Can you verify that the external gpu is visible inside docker?

[KoffeinKaio]

I dont have an integrated GPU, the ARC is the only one available

also when I run your script i can see the process and load via intel_gpu_top

[agrawalsourav98]

Nice. Then the code works for both types of intel GPUs. You got the smart search working right?

[KoffeinKaio]

Im not really sure. I can search for stuff, the ML container loads stuff but the images it displays are not really what I searched for

as I understood smart search i can search for "cars" and it should display photos with them right?

[agrawalsourav98]

Yeah, I think it matches for a lot of things currently and that can be a bit off in certain cases. Like it works like this, you run the model and then match the outputs with a bunch of things, so if the text I am matching with has ["a cat", "a dog", "a bus"], the model can say that 90% cat, 50% dog and 40% bus for an image of a cat. I don't know what filter values are being used but this is how the model works.