[Feature]: Support for Qwen 3 VL

[bhaktatejas922]

🚀 The feature, motivation and pitch

Currently no support for latest generation fo VLMs from Qwen

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[Chenhao-Guan]

Hi team,

I've noticed that Qwen3 (the text model) and Qwen2.5-VL are already supported in the repository. I'm very interested in contributing by adding support for the new Qwen3-VL model.

My current plan is to adapt the existing Qwen2.5-VL implementation as a template, since the architecture seems very similar (ViT + Projector + Qwen3 LLM).

Could you please confirm if this is the correct approach? Is there anything specific I should be aware of?

If this plan looks good, I'd be happy to take on this issue and work on a PR.

Thanks!

[Single430]

Hi team,

I've noticed that Qwen3 (the text model) and Qwen2.5-VL are already supported in the repository. I'm very interested in contributing by adding support for the new Qwen3-VL model.

My current plan is to adapt the existing Qwen2.5-VL implementation as a template, since the architecture seems very similar (ViT + Projector + Qwen3 LLM).

Could you please confirm if this is the correct approach? Is there anything specific I should be aware of?

If this plan looks good, I'd be happy to take on this issue and work on a PR.

Thanks!

Qwen2.5vl to Qwen3-vl model architecture

Interleaved-MRoPE: Full‑frequency allocation over time, width, and height via robust positional embeddings, enhancing long‑horizon video reasoning.

DeepStack: Fuses multi‑level ViT features to capture fine‑grained details and sharpen image–text alignment.

Text–Timestamp Alignment: Moves beyond T‑RoPE to precise, timestamp‑grounded event localization for stronger video temporal modeling.

[karljang]

Hi @yechank-nvidia ,
It seems @Chenhao-Guan is considering contributing support for Qwen3 VL and is asking whether his plan aligns with our recommended practices. Could you share your thoughts on this?

My current plan is to adapt the existing Qwen2.5-VL implementation as a template, since the architecture seems very similar (ViT + Projector + Qwen3 LLM).

[yechank-nvidia]

Hi, @Chenhao-Guan, you are more than welcomed to implement this one.

As @Single430 documented, there are several new features from Qwen2.5-VL.

1. Interleaved-MRope, you might need to extend our MRotaryEmbedding module.
2. DeepStack,You might need to store the ViT's intermediate features and need to transfer to LLM-side.
   Aside from those changes, the other model formats appear to be very similar to the existing Qwen2.5-VL structure.

We would also appreciate it if you could provide a rough estimate of how long you think this implementation might take.

[Chenhao-Guan]

Hi @yechank-nvidia, thanks for the helpful guidance! I'm just starting to get familiar with this repository, so your pointers on Interleaved-MRope and DeepStack are key. Because I'm new to the codebase, it's a bit tough to give a precise timeline, but I'll start digging in and aim to have an initial draft or progress update in [e.g., about 2 weeks]. I also noticed @Single430's detailed breakdown—perhaps we could collaborate on this? I'm excited to get started.

[yechank-nvidia]

Cool. Feel free to tag me when you guys have questions.

[Nekofish-L]

@yechank-nvidia Is it possible to bypass the Python-implemented RoPE calculation within the attention mechanism?

The goal would be to pre-compute the interleaved mRoPE's mrope_rotary_cos_sin and the mrope_position_deltas externally, and then pass these directly into the attention implementation's kernel (e.g., attn_custom_op_inplace). This would allow the kernel to apply the RoPE in a standard way but using our custom interleaved mRoPE parameters.

[yechank-nvidia]

@Nekofish-L, that's what current Qwen2.5-VL is doing. (Ref: https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/models/modeling_qwen2vl.py#L137).

[likebean]

May I ask when support for the Qwen3-VL model will be available?