[Feature Request] datafree pipeline with layer-by-layer quantization

[mratsim]

I am trying to quantize GLM-4.5-Air, GLM-4.5V and finetunes on 2x RTX Pro 6000 to NVFP4A16 (or FP8).

Unfortunately, GLM-4.5-Air with it's 110B parameters requires about 220GB of space and I have 192GB of VRAM.

• The default datafree pipeline fails with Cuda OOM.
• Adding device_map='auto' to AutoModelForCausalLM.from_pretrained still fails, I assume because huggingface/transformers doesn't leave enough space for the extra memory needed during quantization.
• infer_auto_device_map(MODEL_ID, max_memory={0: "85GiB", 1: "85GiB", "cpu": "50GiB"}) fails I think because the model is built differently (error: AttributeError: 'str' object has no attribute 'named_parameters')
• Using pipeline=sequential fails because there is no dataset set.

There might be a way to make it work by explicitly picking which layers go to which GPU and CPU but it's quite cumbersome.

Potential solutions:

1. A calculate_offload_device_map for datafree
2. A datafree_sequential pipeline that does the quantization layer by layer

[dsikka]

Hi @mratsim do you mind sharing your full script? I can take a look.

[mratsim]

Here you go:

from transformers import AutoModelForCausalLM, AutoTokenizer
from accelerate import infer_auto_device_map

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "zai-org/GLM-4.5-Air"
MODEL_OUT = MODEL_ID.split("/")[1] + "-NVFP4A16"

# Load model.
device_map = infer_auto_device_map(MODEL_ID, max_memory={0: "85GiB", 1: "85GiB", "cpu": "50GiB"})
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto", offload_folder = "./offload/", device_map=device_map)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Reference: https://github.com/huggingface/transformers/blob/v4.57.1/src/transformers/models/glm4_moe/modeling_glm4_moe.py
recipe = QuantizationModifier(
    # targets="Linear", F.linear is also used for router_logits so it's better to explicitly include what we want to quantize
    targets = [
        "re:.*gate_proj.*", "re:.*up_proj.*", "re:.*down_proj.*",           # MLP projections
        "re:.*k_proj.*", "re:.*q_proj.*", "re:.*v_proj.*", "re:.*o_proj.*", # Attention projections
    ],
    scheme="NVFP4A16",
    ignore=[
        "lm_head",
        "Glm4MoeRMSNorm",
        "re:.*shared_experts.*",
        "re:.*mlp\\.gate$",             # MoE router
        "re:model\\.layers\\.[0-3]\\.", # Keep first 4 blocks
        "re:.*layers\\.46.*"            # MTP layer (Multi-token prediction)
    ]
)

## References
#
# 1. Why Do Some Inputs Break Low-Bit LLM Quantization? (2025)\
#   Ting-Yun Chang, Muru Zhang, Jesse Thomason, Robin Jia\
#   https://arxiv.org/pdf/2506.12044
#
# 2. Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark (2024)\
#   Pingzhi Li, Xiaolong Jin, Yu Cheng, Tianlong Chen\
#   https://arxiv.org/pdf/2406.08155v1
#
# 3. Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness (2023)\
#   Young Jin Kim, Raffy Fahim, Hany Hassan Awadalla\
#   https://arxiv.org/pdf/2310.02410

oneshot(
    model=model,
    recipe=recipe,
)

# Save to disk in compressed-tensors format.
model.save_pretrained(MODEL_OUT, save_compressed=True)
tokenizer.save_pretrained(MODEL_OUT)
print(f'SUCCESS: files saved in {MODEL_OUT}')

This is for tryout 3 with infer_auto_device_map, the devicemap can be changed to 'auto' for tryout 2.