[RFC] Support new models Qwen3.5-35B-A3B (text input only)

[gali-leilei]

we have a working implementation internally that support Qwen3.5-35B-A3B for text input only:

1. bitwise alignment in forward pass vs transformers==5.2.0.
2. and tested with 8 GPUs with following configuration: (TP, CP) = (1, 1), (2, 1), (1, 2), (2, 2).

Three points I am not sure about:

• multimodal encoder module and MTP module: it is left out for now.
• code organization: torchtitan/models or torchtitan/experiments?
• dependency on flash-linear-attention: it is used for linear attention layer, and tested on both 0.4.0 and 0.4.1.
  • Alternative one is to remove this import. This library is used by slime (ver 0.4.1) and miles (ver 0.4.0) for support Qwen3.5.
  • Alternative two is to guard this import, same treatment as deepep. A pytorch native fallback would still be needed though.

[tianyu-l]

cc @shuhuayu

[gali-leilei]

Draft PR now at #2545

[shuhuayu]

@gali-leilei Sounds great! I think it is nice to have qwen 3.5 in torchtitan as qwen models evolve significantly. And it makes sense to start with a decoder first since it involves most upgrades compared to qwen3 series. I would propose to add it into torchtitan/models as you did in the pr now. Dependencies on flash-linear-attention seems to be necessary at least in the short-term. We currently have an implementation of qwen3 vl under review #2409, and i think much of the vision encoder code and data loading code would be sharable when we evolve qwen3.5 to vlm later.

[gali-leilei]

Dependencies on flash-linear-attention seems to be necessary at least in the short-term.

In that case, should I update pyproject.toml as well?

We currently have an implementation of qwen3 vl under review #2409

Do you want to merge that first into main? I can rebase off it afterwards.

Let me know if there are any other road blockers.

[shuhuayu]

In that case, should I update pyproject.toml as well?

Yeah, i think we should include this dependence into pyproject.toml.

Do you want to merge that first into main? I can rebase off it afterwards.

Thanks for the question. I think for now they can be parallel since the first step is text/decoder only iiuc.

[tianyu-l]

We need to be careful about dependency -- it can be like DeepEP which the code depends on if it's used, but not put in pyproject.toml. Putting it there means everyone who pip install torchtitan would have to depend on FLA, which sounds unnecessary.