[New Model]: XiaomiMiMo/MiMo-Audio-7B-Instruct support

[qibaoyuan]

The model to consider

LLM Model Weights: https://huggingface.co/XiaomiMiMo/MiMo-Audio-7B-Instruct
Audio Tokenizer Weights：https://huggingface.co/XiaomiMiMo/MiMo-Audio-Tokenizer
Model Code: https://github.com/XiaomiMiMo/MiMo-Audio

Model description

This PR enables stage-based deployment for the Mimo-Audio model, aligning it with the vllm-omni architecture. Specific changes include:

Added Stage Configuration

Introduced vllm_omni/model_executor/stage_configs/mimo_audio.yaml to define the multi-stage pipeline.

Refactored Model Structure: Split the Mimo-Audio into two stages:

Stage 0 (LLM+LocalForward)

• Uses MiMoAudioLLMForConditionalGeneration (AR mode) for multimodal understanding and text generation.

Stage 1 (code2wav):

• Uses MiMoAudioToken2WavForConditionalGenerationVLLM for video generation.

Test plan

NOTE: The MIMO_AUDIO_TOKENIZER_PATH environment variable is mandatory due to the specialized architecture.

offline

export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"

python3 -u end2end.py \
  --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
  --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \
  --query-type tts_sft \
  --text "The weather is so nice today."

online

server side:

export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"

vllm-omni serve XiaomiMiMo/MiMo-Audio-7B-Instruct --omni \
--served-model-name "MiMo-Audio-7B-Instruct"  \
--port 8091 --stage-configs-path ./vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
--chat-template ./examples/online_serving/mimo_audio/chat_template.jinja

client:

python openai_chat_completion_client_for_multimodal_generation.py \
--query-type multi_audios \
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json

Test result

TTS(with reference audio)

0_6ca65429-1027-4797-963e-963a1de6c286.wav

Audio understanding

text:

vllm_text_output:
北京今天晴转多云，气温在25到34摄氏度之间，南风三到四级。体感会比较闷热，建议你注意防暑降温，并且尽量避开中午最热的时候出门。另外，紫外线也比较强，记得做好防晒措施。

audio:

0_43fce45f-3809-48f8-b059-351072a4743c.wav

Co-author notice

Please keep the Co-authored-by lines when squashing, as this PR includes contributions from multiple authors (including @Zhang Shijin, @Ding Ning).

Thank you!

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 734774bb95

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[hsliuustc0106]

vllm_omni/worker/gpu_model_runner.py:100

The model loading path should include validation. Add a helpful error message if the model is not found or incompatible.

[princepride]

@qibaoyuan tests/worker/test_omni_gpu_model_runner.py::test_maybe_attach_mimo_audio_req_infos_enriches_dict Unit test failed, PTAL.

[qibaoyuan]

@qibaoyuan tests/worker/test_omni_gpu_model_runner.py::test_maybe_attach_mimo_audio_req_infos_enriches_dict Unit test failed, PTAL.

fixed

[qibaoyuan]

done for #151