is it possible to serve the model using a server (vllm, sglang, etc.)

[AlexiaJM]

See title. By using a server, we can run many inference in parallel, otherwise its very hard to use in agent pipeline.

[AlexiaJM]

Could give an example and specify the version of vllm you used?

This command for example fails on the latest vllm version:

VLLM_USE_V1=0 CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve Qwen/Qwen2.5-Omni-7B
--dtype auto
--max-model-len 32768
--gpu_memory_utilization 0.9
--tensor_parallel_size 4
--port 8001
--mm-processor-kwargs '{"use_audio_in_video": true}'
--host 127.0.0.1 &

---

Using the setup described in the README, it downloads torch 2.6.0, but the vllm version commented out in

Omni-R1/setup.sh

Line 76 in f9a506a

# pip install vllm==0.7.2

requires torch 2.5.1 and the latest version of vllm requires torch 2.7.0 so its not clear what version of vllm you use.

[AlexiaJM]

To add to the issue, QwenThinker was added in vllm version 0.8.5post1 (see https://github.com/vllm-project/vllm/blob/v0.8.5.post1/vllm/model_executor/models/qwen2_5_omni_thinker.py) but this version of vllm requires transformer > 4.51.1 https://github.com/vllm-project/vllm/blob/3015d5634e74d59704e2b39bab0dbe2e6f86a38a/requirements/common.txt#L9 while your code uses a modified version of transformer (4.50.0.dev0) which is incompatible with vllm. 😩

[AlexiaJM]

I would recommend reaching out to the transformer team to make your model compatible with the latest transformer, without this, this model will always be incompatible with newer code and people won't be able to use it. This will also ensure the vllm and other software will handle your model (eventually, because there are still issues with Qwen/Qwen2.5-Omni-7B on the latest vllm as mentioned above).

[AlexiaJM]

Its sadly not possible, because QwenThinker was added in vllm version 0.8.5post1 and this version requires transformer > 4.51.1. More recent versions of vllm requires even higher versions of transformers.

If Omni-R1 can work with the latest transformers (a patch for 4.53.0.dev0), it should be compatible.

Note: Sadly features like audio_in_video are still not working in the latest vllm using 'vllm serve'. But regular features work.