Regarding the Quantization Scheme for the Qwen-Next 80B Model W8A8

[vonchenplus]

Hi there,

I’m planning to quantize the Qwen-Next 80B model to W8A8 using a combination of SmoothQuant and GPTQ. However, I noticed that SmoothQuant does not currently support this model.

In this situation, would it be acceptable to use the default mapping method for SmoothQuant? If not, are there any recommended alternatives or best practices for achieving stable and accurate W8A8 quantization on this model?

Thanks in advance for your help!

[dsikka]

@rahul-tuli please take a look

[rahul-tuli]

Hi @vonchenplus thanks for opening this issue, The SmoothQuant mapping format and how to choose the right modules is documented here in our SmoothQuant Modifier Mapping Tutorial: https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/smoothquant/README.md.
​
This walks through the rationale for targeting q/k/v and feed-forward blocks, explains why we match leaf modules, and shows concrete regex-based mapping examples you can adapt to Qwen-Next 80B.
​
Please take a look and try setting up your mappings based on that guide, and if anything is still unclear or you run into issues with Qwen-Next 80B specifically, let us know here and we’ll be happy to help further.

[vonchenplus]

Hi @vonchenplus thanks for opening this issue, The SmoothQuant mapping format and how to choose the right modules is documented here in our SmoothQuant Modifier Mapping Tutorial: https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/smoothquant/README.md.​This walks through the rationale for targeting q/k/v and feed-forward blocks, explains why we match leaf modules, and shows concrete regex-based mapping examples you can adapt to Qwen-Next 80B.​Please take a look and try setting up your mappings based on that guide, and if anything is still unclear or you run into issues with Qwen-Next 80B specifically, let us know here and we’ll be happy to help further.

Hi, thanks for the detailed explanation and for pointing me to the SmoothQuant mapping tutorial — this is very helpful.

I’ll go through the guide and try setting up the mappings for Qwen-Next 80B accordingly. If I run into any issues or have questions specific to this model, I’ll follow up here.

Thanks again for your help!

[anmarques]

@vonchenplus It turns out you can achieve good results with this model without using SmoothQuant. Here is a model I generated: https://huggingface.co/RedHatAI/Qwen3-Next-80B-A3B-Instruct-quantized.w8a8

@rahul-tuli There is another issue with SmoothQuant here besides mapping. The issue is that this model uses a different implementation of LayerNorm. Namely, they shift the LayerNorm by 1, making it centered at 1 instead of 0:

https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen3_next/modeling_qwen3_next.py#L255

The authors claim this makes training more stable.

The issue is that this breaks the assumptions used to implement the SmoothQuantModifier as is. If this new version of LayerNorm becomes standard we will continue to have problems in the future. The fix is relatively simple. Simply use weight + 1 instead of weight in the balancing calculations. After the SmoothQuant scale is calculated, update the LayerNorm weight as:

weight <- (weight + 1)*scale - 1 

The main challenge is implementing the update in a way that is backwards compatible. I suggest keeping this issue alive until someone implements a resolution.

BTW, the correct mapping for this model can be obtained by this script:

self_attn_layers = list(range(3,49,4))
mappings = []
for layer in self_attn_layers:
    mappings.append([
        [
            f"re:.*layers.{layer}.self_attn.q_proj",
            f"re:.*layers.{layer}.self_attn.k_proj",
            f"re:.*layers.{layer}.self_attn.v_proj",
        ],
        f"re:.*layers.{layer}.input_layernorm"]
    )

[brian-dellabetta]

@rahul-tuli There is another issue with SmoothQuant here besides mapping. The issue is that this model uses a different implementation of LayerNorm. Namely, they shift the LayerNorm by 1, making it centered at 1 instead of 0:

https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen3_next/modeling_qwen3_next.py#L255

The authors claim this makes training more stable.

The issue is that this breaks the assumptions used to implement the SmoothQuantModifier as is. If this new version of LayerNorm becomes standard we will continue to have problems in the future. The fix is relatively simple. Simply use weight + 1 instead of weight in the balancing calculations. After the SmoothQuant scale is calculated, update the LayerNorm weight as:

weight <- (weight + 1)*scale - 1 

The main challenge is implementing the update in a way that is backwards compatible. I suggest keeping this issue alive until someone implements a resolution.

This will be resolved by #2500