[OpenVINO]add support for minicpmv4/4_5 #1412

openvino-dev-samples · 2025-08-07T03:08:15Z

Depends on PR

As LLM of minicpmv4 switched to Llama
https://huggingface.co/openbmb/MiniCPM-V-4/blob/main/modeling_minicpmv.py#L26

What does this PR do?

Conversion cmd-line for openbmb/MiniCPM-V-4 or MiniCPM-V-4_5:

optimum-cli export openvino --model openbmb/MiniCPM-V-4_5 MiniCPM-V-4_5-ov --trust-remote-code --weight-format fp16 --task image-text-to-text

Inference of MiniCPM-V-4_5 using OpenVINO backend:

from optimum.intel.openvino import OVModelForVisualCausalLM
from transformers import AutoProcessor
from PIL import Image
import requests

model_id = "openbmb/MiniCPM-V-4_5"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

prompt= "<|im_start|>user\n(<image>./</image>)\nWhat is unusual on this image?<|im_end|>\n<|im_start|>assistant\n"
image = Image.open(requests.get("https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/d5fbbd1a-d484-415c-88cb-9986625b7b11", stream=True).raw).convert('RGB')

model = OVModelForVisualCausalLM.from_pretrained(model_dir, trust_remote_code=True)

inputs = processor([prompt], [image], return_tensors="pt")

result  = model.generate(**inputs, max_new_tokens=20)

print(processor.tokenizer.batch_decode(result[:, inputs["input_ids"].shape[1]:]))

Before submitting

Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

HuggingFaceDocBuilderDev · 2025-08-07T07:22:55Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

openvino-dev-samples · 2025-08-08T09:27:55Z

@IlyasMoutawwakil could help to take a look ?

IlyasMoutawwakil · 2025-08-08T09:56:36Z

Thanks for the fix ! let's create a tiny random model with llama as the decoder to test this 🤗 tell me you need help with that !

openvino-dev-samples · 2025-08-08T11:04:18Z

Thanks for the fix ! let's create a tiny random model with llama as the decoder to test this 🤗 tell me you need help with that !

But i guess we need merge this PR first ? otherwise test case will not work

IlyasMoutawwakil · 2025-08-08T12:52:42Z

@openvino-dev-samples no need to merge it now, you can simply pin that PR in setup.py so that the tests would run with it 🤗
We will merge both PRs once everything works together.

openvino-dev-samples · 2025-08-12T00:38:03Z

@openvino-dev-samples no need to merge it now, you can simply pin that PR in setup.py so that the tests would run with it 🤗 We will merge both PRs once everything works together.

Hi since minicpmv4 and minicpmv share a same model type, but different LLM. It is possible to add both of them in utils_tests.py ?

IlyasMoutawwakil · 2025-08-12T08:00:35Z

@openvino-dev-samples yes, you can name it minicpmv4 in utils_tests.py

IlyasMoutawwakil · 2025-08-18T18:31:19Z

Hi @openvino-dev-samples it would be faster if you made sure the minicpmv4 tests pass locally, the ci is slow and shouldn't be used as a testing mechanism, only use it for validating when local tests are already passing.

openvino-dev-samples · 2025-08-19T01:14:05Z

Hi @openvino-dev-samples it would be faster if you made sure the minicpmv4 tests pass locally, the ci is slow and shouldn't be used as a testing mechanism, only use it for validating when local tests are already passing.

Sorry for that, and i fully understand, but i always met connection issue in local test case, e.g

huggingface_hub.errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/models/katuni4ka/tiny-random-qwen2.5-vl/tree/main?recursive=True&expand=False

IlyasMoutawwakil · 2025-08-19T08:33:16Z

but i always met connection issue in local test case

you can target minicpmv tests specifically to avoid this issue with pytest -k "minicpmv"

rkazants · 2025-10-13T03:18:15Z

tests/openvino/utils_tests.py

    "minicpm3": "katuni4ka/tiny-random-minicpm3",
    "minicpmv": "katuni4ka/tiny-random-minicpmv-2_6",
+    "minicpmv4": "snake7gun/minicpm-v-4-tiny",
+    "minicpmv4_5": "snake7gun/tiny-minicpmv-4_5",


158M model size, it makes sense to try to reduce the size

rkazants

please add tests for inference to test generate() method and compare with transformers

openvino-dev-samples · 2025-10-21T02:27:02Z

@IlyasMoutawwakil could you help to trigger CI, thanks

IlyasMoutawwakil · 2025-10-22T07:30:59Z

optimum/exporters/openvino/model_configs.py

        if isinstance(behavior, str) and not isinstance(behavior, MiniCPMVConfigBehavior):
            behavior = MiniCPMVConfigBehavior(behavior)
-
+        model_mapping = {2.6: "llama", 4.0: "qwen2", 4.5: "qwen3"}


should use str for versions

may i understand why？the version in model's config is a number:
https://huggingface.co/openbmb/MiniCPM-V-4_5/blob/main/config.json#L3

ah okay I see ! thanks for the clarification.
(it's generally a bad idea to use numbers for versions: 4.0 becomes 4 and 4.10 and 4.1 are the same version 😅)

rkazants · 2025-10-26T11:10:16Z

optimum/exporters/openvino/model_configs.py

        if isinstance(behavior, str) and not isinstance(behavior, MiniCPMVConfigBehavior):
            behavior = MiniCPMVConfigBehavior(behavior)
-
+        model_mapping = {2.6: "llama", 4.0: "qwen2", 4.5: "qwen3"}


I think it is a bad idea to make decision about architecture based on model version in general.
I think you should parse class model object and use isinstance for inner objects to make decision.

Yes, its a better approach in this case, but I dont know if we can access the modeling file at this stage.

tests/openvino/test_seq2seq.py

fix CI

nikita-savelyevv · 2025-11-01T12:11:38Z

@openvino-dev-samples Please fix the failed tests

openvino-dev-samples · 2025-11-03T00:41:01Z

@openvino-dev-samples Please fix the failed tests

done

rkazants

please do additional patch for temporal_ids as we discussed. Without this, functionality is limited.

nikita-savelyevv

Please also update the PR description according to this comment #1491 (comment)