[Quant] Support MXFP4 W4A16 for compressed-tensors MoE models by dsikka · Pull Request #138 · neuralmagic/vllm

dsikka · 2026-01-09T23:57:12Z

Purpose

Follow-up to: vllm-project#31926
Add MoE support for MXFP4 CT Models. Currently only supporting weight-only quantization through the marlin integration.

There's a ton of shared boilerplate code between mxfp4 and nvfp4 that we should consider combining.

Test Plan

Qwen3 MoE MXFP4A16: https://huggingface.co/nm-testing/Qwen3-30B-A3B-MXFP4A16/tree/main

Test Result

Running GSM8K evaluation: 1319 questions, 5-shot
Evaluating: 100%|█████████████████████████████| 1319/1319 [00:29<00:00, 45.47it/s]

Results:
Accuracy: 0.882
Invalid responses: 0.000
Total latency: 29.023 s
Questions per second: 45.446
Total output tokens: 156720
Output tokens per second: 5399.780

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: Jeremy Teboul <jeremyte@meta.com>

…ject#29354) Signed-off-by: Runkai Tao <rt572@physics.rutgers.edu> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

…llm-project#31836) Signed-off-by: Andrew Xia <axia@fb.com> Co-authored-by: Andrew Xia <axia@fb.com>

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

…nd kv_layout (vllm-project#30275)

…-project#31336) Signed-off-by: izhuhaoran <izhuhaoran@qq.com> Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>

…oject#31998) Signed-off-by: Nick Hill <nickhill123@gmail.com>

mgoin · 2026-01-10T00:01:34Z

vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py

+
+        self.moe_quant_config = self.get_fused_moe_quant_config(layer)
+        if self.moe_quant_config is not None:
+            self.kernel = make_nvfp4_moe_kernel(


This works because Marlin autodetects mxfp4?

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

Signed-off-by: Russell Bryant <rbryant@redhat.com>

…t#32056) Signed-off-by: yewentao256 <zhyanwentao@126.com>

) Signed-off-by: Micah Williamson <micah.williamson@amd.com>

…t#31616) Signed-off-by: c0de128 <kevin.mckay@outlook.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Signed-off-by: Andy Xie <andy.xning@gmail.com>

Signed-off-by: Lucas Kabela <lucaskabela@meta.com>

Signed-off-by: Roger Wang <hey@rogerw.io> Co-authored-by: Roger Wang <hey@rogerw.io>

Signed-off-by: Patryk Saffer <patryk.saffer99@gmail.com> Signed-off-by: PatrykSaffer <patryk.saffer@mistral.ai> Co-authored-by: Patryk Saffer <patryk.saffer99@gmail.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

…rocess (vllm-project#31295) Signed-off-by: c0de128 <kevin.mckay@outlook.com>

…imize forward (vllm-project#31939) Signed-off-by: maang <maang_h@163.com>

) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…project#31895) Signed-off-by: Roi Koren <roik@nvidia.com>

…chema test examples (vllm-project#32063) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

…project#32074) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

…vision encoder (vllm-project#32233) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

Signed-off-by: Xingyu Liu <charlotteliu12x@gmail.com>

Signed-off-by: esmeetu <jasonailu87@gmail.com>

vllm-project#32061) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

vllm-project#32126) Signed-off-by: YunzhuLu <lucia.yunzhu@gmail.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…m-project#32243) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…ct#32198) Signed-off-by: NickLucche <nlucches@redhat.com>

…ject#29867) Signed-off-by: Mickael Seznec <mickael@mistral.ai> Signed-off-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

…architecture (vllm-project#32240) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

…ct#32215) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…-project#32251) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

…ect#32169) Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…ory leak (vllm-project#32181) Signed-off-by: Mathis Felardos <mathis@mistral.ai>

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

Signed-off-by: Mark McLoughlin <markmc@redhat.com>

…-project#32050) Signed-off-by: Sage Moore <sage@neuralmagic.com>

Signed-off-by: Andrew Xia <axia@fb.com> Signed-off-by: Andrew Xia <mitandrewxia@gmail.com> Signed-off-by: Lu Fang <30275821+houseroad@users.noreply.github.com> Co-authored-by: Andrew Xia <axia@fb.com> Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>

Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

…nt (vllm-project#32058) Signed-off-by: yewentao256 <zhyanwentao@126.com>

…vllm-project#31711) Signed-off-by: rabi <ramishra@redhat.com>

Signed-off-by: Dipika Sikka <dipikasikka1@gmail.com>

jeremyteboul and others added 10 commits January 9, 2026 19:34

[Fix] Introduce audio channels spec (vllm-project#31595)

657e9c0

Signed-off-by: Jeremy Teboul <jeremyte@meta.com>

Add unpermute-aware fused MoE path and small-batch fallback (vllm-pro…

a4d5d66

…ject#29354) Signed-off-by: Runkai Tao <rt572@physics.rutgers.edu> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

[fix] add cutedsl to global sf (vllm-project#32001)

f9e2a75

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>

[Quant] Make static quant support all group shapes (vllm-project#30833)

0a0aa07

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

[responsesAPI] fix incomplete_messages for simple/parsable context (v…

1f8b7c5

…llm-project#31836) Signed-off-by: Andrew Xia <axia@fb.com> Co-authored-by: Andrew Xia <axia@fb.com>

[1/N][Attention] Restructure attention: move files (vllm-project#31916)

2612ba9

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

[NIXL] refine decoder side post process for heterogeneous BlockSize a…

9457812

…nd kv_layout (vllm-project#30275)

[perf][async] support non cpu sync get logprob tensors for spec (vllm…

97ba96f

…-project#31336) Signed-off-by: izhuhaoran <izhuhaoran@qq.com> Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>

[Misc] Enable async scheduling by default with spec decoding (vllm-pr…

3adffd5

…oject#31998) Signed-off-by: Nick Hill <nickhill123@gmail.com>

[Misc][BE] Type coverage for vllm/compilation [2/3] (vllm-project#31744)

aaf4b70

dsikka requested review from mgoin, robertgshaw2-redhat, tlrmchlsmth and yewentao256 as code owners January 9, 2026 23:57

mgoin reviewed Jan 10, 2026

View reviewed changes

MatthewBonanni and others added 15 commits January 10, 2026 00:27

[2/N][Attention] Fix pre-commit errors (vllm-project#32052)

0308901

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

[Core] Use weights_only=True with torch.load (vllm-project#32045)

1963245

Signed-off-by: Russell Bryant <rbryant@redhat.com>

[Perf] Optimize async scheduling placeholder using empty (vllm-projec…

e18464a

…t#32056) Signed-off-by: yewentao256 <zhyanwentao@126.com>

[CI] Allow Deprecated Quantization For LM Eval Tests (vllm-project#32065

ac0675f

) Signed-off-by: Micah Williamson <micah.williamson@amd.com>

[Bugfix] Narrow broad exceptions in compilation backends (vllm-projec…

4dc0d60

…t#31616) Signed-off-by: c0de128 <kevin.mckay@outlook.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

resolve pydantic error in startup benchmark (vllm-project#31348)

abd9224

Signed-off-by: Andy Xie <andy.xning@gmail.com>

[Misc][LLaMa4] Compile LLaMa Vision Encoder (vllm-project#30709)

ea6d067

Signed-off-by: Lucas Kabela <lucaskabela@meta.com>

feature/issac 0.2 (vllm-project#31550)

e45946b

Signed-off-by: Roger Wang <hey@rogerw.io> Co-authored-by: Roger Wang <hey@rogerw.io>

[Bugfix][Hardware][AMD] Use dynamic WARP_SIZE in sampler vectorized_p…

c60578d

…rocess (vllm-project#31295) Signed-off-by: c0de128 <kevin.mckay@outlook.com>

[Core] Refactor ColumnParallelLinear: remove unused parameter and opt…

52d4282

…imize forward (vllm-project#31939) Signed-off-by: maang <maang_h@163.com>

[Refactor] Separate sequence and token pooling types (vllm-project#32026

583a90e

) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Update modelopt KV cache quantization resolution to new scheme (vllm-…

0c96148

…project#31895) Signed-off-by: Roi Koren <roik@nvidia.com>

[ROCm][CI] Fix flaky test_function_calling_with_stream and reduce s…

d83becd

…chema test examples (vllm-project#32063) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

[Misc] Delay deprecation of CommonAttentionMetadata properties (vllm-…

da6709c

…project#32074) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

AndreasKaratzas and others added 22 commits January 12, 2026 22:33

[ROCm][CI] Fix HuggingFace flash_attention_2 accuracy issue in Isaac …

5e714f7

…vision encoder (vllm-project#32233) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

[BugFix]Fix eagle draft_model_config and add tests (vllm-project#31753)

80221e1

Signed-off-by: Xingyu Liu <charlotteliu12x@gmail.com>

[Doc] Update installation from source command (vllm-project#32239)

44c34f2

Signed-off-by: esmeetu <jasonailu87@gmail.com>

[ROCm][CI] Fix engine core client tests for ROCm spawn multiprocessing (

df7e127

vllm-project#32061) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

[Model] Use mm_position to compute mrope positions for Qwen2-VL/2.5-VL (

542a405

vllm-project#32126) Signed-off-by: YunzhuLu <lucia.yunzhu@gmail.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

[Refactor] Remove get_encoder_dummy_data (vllm-project#32241)

eb28e80

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Bugfix] Replace PoolingParams.normalize with use_activation (vll…

232214b

…m-project#32243) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Docs] Nixl Usage recommend fail kv_load_failure_policy (vllm-proje…

8c8653b

…ct#32198) Signed-off-by: NickLucche <nlucches@redhat.com>

[Refactor] [6/N] to simplify the vLLM openai chat_completion serving …

fefce49

…architecture (vllm-project#32240) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

[6/N][Attention] Move utils to more appropriate locations (vllm-proje…

98f60e5

…ct#32215) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

[Refactor] Remove MultiModalProfiler (vllm-project#32254)

252c011

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Refactor] [7/N] to simplify the vLLM lora serving architecture (vllm…

4f02cb2

…-project#32251) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

[BugFix] [KVConnector] Fix KV events for LMCache connector (vllm-proj…

5102654

…ect#32169) Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

nixl_connector: export UCX_MEM_MMAP_HOOK_MODE=none to avoid a UCX mem…

4f3676e

…ory leak (vllm-project#32181) Signed-off-by: Mathis Felardos <mathis@mistral.ai>

[4/N][Attention] Move MLA common to model_executor (vllm-project#32060)

2263d44

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

[Trivial] Remove duplicate enable_mfu_metrics (vllm-project#32246)

ab74b2a

Signed-off-by: Mark McLoughlin <markmc@redhat.com>

[EPLB][Cleanup] Remove is_async_enabled from EplbModelState (vllm…

6beef12

…-project#32050) Signed-off-by: Sage Moore <sage@neuralmagic.com>

Fix CUDA 13 wheel installation doc (vllm-project#32276)

46f8c6b

Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

[Perf] Optimize grouped topk kernel, 1.2%~2% E2E Throughput improveme…

f28125d

…nt (vllm-project#32058) Signed-off-by: yewentao256 <zhyanwentao@126.com>

fix(rocm): Use refresh_env_variables() for rocm_aiter_ops in test_moe (…

69f8a0e

…vllm-project#31711) Signed-off-by: rabi <ramishra@redhat.com>

dsikka requested review from LucasWilkinson, NickLucche, ProExpertProg, alexm-redhat and youkaichao as code owners January 13, 2026 19:46

add mxfp4 ct moe support

af677f5

Signed-off-by: Dipika Sikka <dipikasikka1@gmail.com>

dsikka force-pushed the mxfp4-w4a16-ct-moe branch from eec67ec to af677f5 Compare January 13, 2026 19:51

dsikka closed this Jan 13, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Quant] Support MXFP4 W4A16 for compressed-tensors MoE models #138

[Quant] Support MXFP4 W4A16 for compressed-tensors MoE models #138
dsikka wants to merge 122 commits intomxfp4-w4a16-ct-densefrom
mxfp4-w4a16-ct-moe

dsikka commented Jan 9, 2026 •

edited by github-actions bot

Loading

Uh oh!

mgoin Jan 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

dsikka commented Jan 9, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

mgoin Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

dsikka commented Jan 9, 2026 •

edited by github-actions bot

Loading