feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron#2304

Merged

yzh119 merged 25 commits intoflashinfer-ai:mainfrom

amitz-nv:fused-moe-non-gated-fp8

Jan 30, 2026

Commits on Jan 28, 2026

Using ActivationType instead of GatedActType, added compiled kernels, adjusted test
amitz-nv
committed
Add actType and eltwiseActType to 'no kernel found' message, move is_gated_activation function in tests to tests/moe/utils.py
amitz-nv
committed
Update remaining GatedActType uses to ActivationType, remove GatedActType enum from core.py, update docstrings
amitz-nv
committed
Use ActivationType in benchmarks, add missing activation_type argument
amitz-nv
committed
Minor fixes
amitz-nv
committed
Fix activation_type default value to Swiglu on trtllm_fp4_block_scale_moe
amitz-nv
committed
Minor improvement
amitz-nv
committed
Support non-gated activation in NVFP4 block scale MoE
amitz-nv
committed
Rename useShuffledMatrixA to useShuffledMatrix (remove the 'A' suffix)
amitz-nv
committed
Add FP4_NVFP4_NVFP4 parameterization to test_llama4_routing, update tests skip_checks to skip on non-gated activation with quantizations that don't support it
amitz-nv
committed
Increase supported topK and num experts in deepseek routing for nemotron
amitz-nv
committed
Commit more files for increase supported topK and num experts in deepseek routing for nemotron
amitz-nv
committed
Fix formatting
amitz-nv
committed
Change TODO to comment
amitz-nv
committed
Change default activation_type to Swiglu
amitz-nv
committed
Restore intermediate size factor of 2 for gated activation in getWorkspaceSizeInBytes, getDefaultValidConfigIndex, isValidConfigIndex
amitz-nv
committed
Formatting fixes
amitz-nv
committed
Treat SwigluBias as gated activation
amitz-nv
committed
Fix use of ActivationType enum in CLI
amitz-nv
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron#2304

feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron#2304
yzh119 merged 25 commits intoflashinfer-ai:mainfrom
amitz-nv:fused-moe-non-gated-fp8

Commits on Jan 28, 2026

Using ActivationType instead of GatedActType, added compiled kernels, adjusted test

Add actType and eltwiseActType to 'no kernel found' message, move is_gated_activation function in tests to tests/moe/utils.py

Update remaining GatedActType uses to ActivationType, remove GatedActType enum from core.py, update docstrings

Use ActivationType in benchmarks, add missing activation_type argument

Minor fixes

Fix activation_type default value to Swiglu on trtllm_fp4_block_scale_moe

Minor improvement

Support non-gated activation in NVFP4 block scale MoE

Rename useShuffledMatrixA to useShuffledMatrix (remove the 'A' suffix)

Add FP4_NVFP4_NVFP4 parameterization to test_llama4_routing, update tests skip_checks to skip on non-gated activation with quantizations that don't support it

Increase supported topK and num experts in deepseek routing for nemotron

Commit more files for increase supported topK and num experts in deepseek routing for nemotron

Fix formatting

Change TODO to comment

Change default activation_type to Swiglu

Restore intermediate size factor of 2 for gated activation in getWorkspaceSizeInBytes, getDefaultValidConfigIndex, isValidConfigIndex

Formatting fixes

Treat SwigluBias as gated activation

Fix use of ActivationType enum in CLI

Commits on Jan 29, 2026

Fix activation-type command line argument handling in benchmarks

Fix choices of activation-type command line argument handling in benchmarks

GEMM (non batched) still has mUseShuffledMatrixA member (with 'A' suffix)

Update bench_trtllm_gen_fused_moe_autotuner.py to support more activations

Revert activation_Type check in bench_trtllm_gen_fused_moe_autotuner.py for trtllm_fp8_block_scale_moe

Include activation type in results in benchmarks/routings/moe.py