[Feature Request] Qwen3VL GRPO, SFT training

[eagle705]

Additional context

Our customer would like to apply RL methods (GRPO, GSPO, and SPO) to VLM with MoE (such as Qwen3-VL).

Would it be possible to extend the current VLM support to Qwen3-VL?

(cc. @terrykong, @snowmanwwg )

[euronymous-aithal]

@yfw please opine

[euronymous-aithal]

updating the status here from @yfw : This seems like a large model so we will most likely need to use mcore path for this. We recently just merged this PR for VLM + mcore support for Qwen2.5-VL: #1115
Supporting this new model in NeMo-RL will require adding Qwen3 VL in Megatron-Bridge. @yaoyu-33 is already working on this: NVIDIA-NeMo/Megatron-Bridge#775. this is actually the main change. Once we have support for this model in Megatron-Bridge, we will just have to test it in NeMo-RL.
assigning this task to @eagle705

[eagle705]

Current issues about Qwen3-VL-30B-A3B (run_vlm_grpo.py)

• vllm version: [Usage]: ValueError: The checkpoint you are trying to load has model type qwen3_moe but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date. vllm-project/vllm#19793
  • log

�[36m(VllmGenerationWorker pid=3212593, ip=10.52.54.131)�[0m   File "/opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker.VllmGenerationWorker/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1273, in from_pretrained
�[36m(VllmGenerationWorker pid=3212593, ip=10.52.54.131)�[0m     raise ValueError(
�[36m(VllmGenerationWorker pid=3212593, ip=10.52.54.131)�[0m ValueError: The checkpoint you are trying to load has model type `qwen3_vl_moe` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

• fast tokenizer
  • log

Traceback (most recent call last):
  File "/work/code/RL/examples/run_vlm_grpo.py", line 335, in main
    processor = get_tokenizer(config["policy"]["tokenizer"], get_processor=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/nemo-rl/nemo_rl/algorithms/utils.py", line 264, in get_tokenizer
    tokenizer = processor.tokenizer
                ^^^^^^^^^^^^^^^^^^^
  File "/opt/nemo_rl_venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1099, in __getattr__
    raise AttributeError(f"{self.__class__.__name__} has no attribute {key}")
AttributeError: Qwen2TokenizerFast has no attribute tokenizer. Did you mean: '_tokenizer'?

[therealnaveenkamal]

Hi @snowmanwwg / @eagle705 do you think I can help here in any way?

[eagle705]

@snowmanwwg @euronymous-aithal
I just confirmed that Qwen3-VL and Qwen3-VL MoE work well when sequence packing (THD format) is enabled (cc: @wplf)

  policy.megatron_cfg.apply_rope_fusion=False \
  policy.sequence_packing.enabled=True \

• Example script

NRL_FORCE_REBUILD_VENVS=true uv run python examples/run_vlm_grpo.py \
  --config /opt/nemo-rl/examples/configs/vlm_grpo_3B_megatron.yaml \
  cluster.gpus_per_node=8 \
  policy.model_name="Qwen/Qwen3-VL-4B-Instruct" \
  policy.dynamic_batching.enabled=False \
  policy.megatron_cfg.apply_rope_fusion=False \
  policy.sequence_packing.enabled=True \
  checkpointing.checkpoint_dir="results/vlm_grpo_4B_megatron" \
  logger.wandb_enabled=True \
  logger.wandb.name="vlm_grpo_qwen3vl_4B_megatron" \
  logger.num_val_samples_to_print=10

If we want to use it without sequence packing, we need to fix the get_rope_index arguments by properly handling the attention_mask, which is not currently None.

• Reference: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/e81e80ef9b8acf74714f339d73c6a872b270eaed/src/megatron/bridge/models/qwen_vl/modeling_qwen25_vl.py#L198-L205

Without sequence packing, the following error occurred.

  File "/work/code/nemo-sa/RL/nemo-rl/3rdparty/Megatron-Bridge-workspace/Megatron-Bridge/src/megatron/bridge/models/qwen_vl/modelling_qwen3_vl/model.py", line 457, in forward
    position_ids, _ = get_rope_index(
                      ^^^^^^^^^^^^^^^
  File "/work/code/nemo-sa/RL/nemo-rl/3rdparty/Megatron-Bridge-workspace/Megatron-Bridge/src/megatron/bridge/models/qwen_vl/modelling_qwen3_vl/rope.py", line 188, in get_rope_index
    sample_input_ids = sample_input_ids[attention_mask[i] == 1]
                       ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: too many indices for tensor of dimension 1

Here is a PR for this case

• fix: use HF-style attn mask path for qwen3-vl RoPE in BSHD training Megatron-Bridge#2582