Skip to content

[LLM:Feature] Add Qwen3.5 video input#4577

Open
Fatemanx wants to merge 1 commit into
alibaba:masterfrom
Fatemanx:feature/qwen35-video-input
Open

[LLM:Feature] Add Qwen3.5 video input#4577
Fatemanx wants to merge 1 commit into
alibaba:masterfrom
Fatemanx:feature/qwen35-video-input

Conversation

@Fatemanx

@Fatemanx Fatemanx commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Add Qwen3.5/Qwen-style video prompt support in the LLM Omni path.

This implements video decoding through OpenCV videoio, HF-style frame sampling, timestamped video prompts, video pad token replacement, temporal M-RoPE positions, and block-diagonal visual attention masks for Qwen video semantics.

Key Changes

  • Add optional MNN_LLM_VIDEOIO_OPENCV support for video prompt decoding.
  • Support <video>...</video> prompts with sampled video frames and timestamp groups.
  • Use video_pad embeddings separately from image pad embeddings.
  • Align Qwen3.5 video preprocessing with HF-style sampling and temporal visual attention.
  • Add Python and C++ regression tests for video prompt semantics, sampling, and mask behavior.

Validation

  • git diff --check --cached
  • clang-format --dry-run --Werror ...
  • python transformers/llm/export/test_qwen35_video_semantics.py
  • cmake --build build_qwen35_video_e2e --target llm_demo test_omni_video_prompt -j$(nproc)
  • build_qwen35_video_e2e/test_omni_video_prompt
  • E2E generated 1s/25fps number videos:
    • forward 1..25 -> increasing
    • reverse 25..1 -> decreasing

@Fatemanx
Fatemanx marked this pull request as ready for review June 28, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant