[QWen3_VL] pretrain performance optimization

[shifangx]

QWen3-VL pretrain performance optimization

This issue is used to track QWen3-VL pretrain performance optimization.

Functional Support

• model support
• • Add Qwen3VL support (dense and moe) #1174
• • Add example for Qwen3-VL-235B-A22B finetune #1533
• • QWen3-VL Performance Test #1769
• various image resolution and text length
• • [to be done]
• sequence packing
• • [to be done]
• other issues
• • a temporary fix for pipeline parallelism, to fix [QWen3_VL] support pipeline parallelism, by broadcast input_ids. #1631
• • [QWen3-VL] It is seams that all of dp ranks get the same sample. #1606
• • Do we need to split the vision_embeds? Add Qwen3VL support (dense and moe) #1174 (comment)
• support video in dataset samples
• • [to be done]

Parallel optimization

• Baseline
• • PP+EP, Encoder on the first pp stage.
• M-FSDP
• • M-FSDP for both Vision model and LLM backbone model
• DistTrain
• • [QWen3_VL] [DistTrain] Disaggregated training system #1632
• MDP
• • [MLLM] Modality-Decoupled Parallelism(MDP) #1554
• Integrate MDP/DistTrain with other features
• • Intergrate with M-FSDP. For example, use M-FSDP for Encoder, and use other 3-D parallelism for LLM backbone.
• • Integrate with Hybrid CP or MagiAttention
• • Integrate DistTrain with interleaved 1F1B overlap, which is essential for EP's performance.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.