[Qwen3.5] Calibration support and NVFP4 Example

[dsikka]

SUMMARY:

• Add Qwen3.5 modeling support and an nvfp4 example
• Specifically unstack 3D weights
• Handle mlp.gates being Parameters in the original definition and therefore not originally added to the ignore list
• This model has some dead weights and therefore requires: Update generate_gparam calculation to handle NaNs and Infs compressed-tensors#637
• Updates some functionality to be compatible with transformers v5, which is required for this model. You will need to update to transformers v5 to use this example

Closes: #2458

Checkpoint:

https://huggingface.co/RedHatAI/Qwen3.5-122B-A10B-NVFP4

Next Steps:

• Add a step to include MTP layers to the final checkpoint which are in the original checkpoint but not loaded through the AutoModel pathway

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates support for the Qwen3.5 Mixture-of-Experts (MoE) model, enabling its quantization using the NVFP4 scheme. It includes a new example demonstrating the end-to-end quantization process, from model loading and dataset preparation to applying the quantization recipe and saving the compressed model. Key architectural adjustments were made to handle Qwen3.5's specific weight structures and ensure compatibility with the latest transformers library, facilitating efficient and accurate model compression.

Highlights

• Qwen3.5 Modeling Support: Added comprehensive support for the Qwen3.5 Mixture-of-Experts (MoE) model.
• NVFP4 Quantization Example: Introduced a new example demonstrating NVFP4 quantization for the Qwen3.5 model.
• Weight Unstacking: Implemented logic to correctly unstack 3D weights specific to Qwen3.5 experts, enabling individual quantization.
• Parameter Handling: Addressed the handling of mlp.gates which are defined as Parameters in the original Qwen3.5 model, ensuring they are properly managed during quantization.
• Transformers v5 Compatibility: Updated core functionalities to maintain compatibility with the transformers library version 5.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• examples/quantization_w4a4_fp4/qwen3_5_example.py
  • Added a new example script for NVFP4 quantization of the Qwen3.5 model.
  • Configured QuantizationModifier to target "Linear" layers with "NVFP4" scheme, specifying layers to ignore.
  • Included dataset loading and preprocessing steps for calibration using ultrachat_200k.
  • Applied oneshot quantization with moe_calibrate_all_experts enabled.
  • Implemented saving of the quantized model and processor.
• src/llmcompressor/modeling/init.py
  • Imported CalibrationQwen3_5MoeSparseMoeBlock to register the new Qwen3.5 MoE calibration module.
• src/llmcompressor/modeling/qwen3_5_moe.py
  • Added CalibrationQwen3_5MoeSparseMoeBlock to unfuse 3D expert parameters into individual nn.Linear modules for quantization.
  • Implemented SequentialQwen3_5MoeExperts to correctly handle and unfuse the Qwen3.5 MoE expert weights.
• src/llmcompressor/utils/dev.py
  • Modified TORCH_INIT_FUNCTIONS import to be compatible with both transformers versions prior to v5 and v5+.
• src/llmcompressor/utils/pytorch/module.py
  • Updated the get_no_split_params function to handle no_split_modules retrieval for both transformers versions prior to v5 and v5+.

Activity

• No activity to report yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds calibration support for the Qwen3.5 model and includes an NVFP4 quantization example. The changes introduce a new calibration module for Qwen3.5 MoE layers, an example script demonstrating its use, and several compatibility updates for transformers v5. My review identified a few issues, including a hardcoded configuration that limits functionality, a significant performance issue in the new calibration logic, a misleading comment in the example script, and a minor code style inconsistency. I have provided specific feedback and code suggestions to address these points.

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[phaelon74]

Thanks for snagging and getting this added @dsikka . Your note about next steps, is that still valid to MTP.

I also wonder how this will hold up when Quanting a non MOE Qwen3.5. Today, QWEN3.5-27B Text only has several bugs in both vllm and llm_compressor, where it behaves weirdly. I surmise this will not solve those, but I might try to requant with this PR. Thanks again!

[dsikka]

Thanks for snagging and getting this added @dsikka . Your note about next steps, is that still valid to MTP.

I also wonder how this will hold up when Quanting a non MOE Qwen3.5. Today, QWEN3.5-27B Text only has several bugs in both vllm and llm_compressor, where it behaves weirdly. I surmise this will not solve those, but I might try to requant with this PR. Thanks again!

Thanks for bringing this up! Another community member put up an example PR for this: #2467

If you have any specific issues, do you mind opening an issue and we can investigate?