Roadmap (2025 Q4)

[sricketts]

We would like feedback from the community on this rough plan for Q4. This is of course a work in progress, and we welcome feedback at any time. Please add comments below or on any specific issues. We will edit this description as plans change.

Also, if you're interested in contributing, feel free to dive into any of the unassigned issues!

Soul

These are the broad areas of focus for the quarter. Items in the roadmap below are tagged by “soul item”.

• [Testing]: Better testing and CI infrastructure to prevent build breaks and accuracy issues at the framework level
• [Model Optimization]: DeepSeek-R1, GPT-OSS, Qwen3, Qwen3-Next, MiniCPM4.1-8B, and others
• [API Usability]: API cleanup and refactoring for better user experience

October

• [Model Optimization] DSR1 improvements (details TBD)
• [Model Optimization] Update the routing for TRTLLMGEN to support kimi k2 and qwen #1831
• [Model Optimization] [Feature Request] Gated Delta Net #1690
• [Model Optimization] GPT-OSS perf improvements for max throughput case
• [Model Optimization] Native Sparse Attention
• [Model Optimization] [Feature Request] TopK Sparse Attention #1691
• [API Usability] [Feature Request] "auto" backend for mm_fp4 #1722
• [Testing] Expanded CI coverage per-PR (ability to trigger tests on NVIDIA-internal test infrastructure, including various Blackwell devices)
• [Testing] Initial integration testing: e2e functional sanity checks
• [Testing] Add comprehensive test xfails tracking system and analysis report #1733
• [Model Optimization] Non-gated MoE with squared ReLU activation
• [Model Optimization] [Perf] FP4 MoE on B200 (latency) #1734
• [Model Optimization] [Perf] FP4 GEMM on B200 #1732
• [Model Optimization] Add RoPE, RoPE+Q, RoPE+Q+KVCacheUpdate fused kernels for MLA/GQA/MHA
• [API Usability] Uniform behavior when a (backend, target device, problem shape) is not supported
• [API Usability] More clear specification of SM func and perf support across interfaces/backends.
• [API Usability] refactor: using tvm-ffi for multi-platform bindings #1641
• [API Usability] Unifying quantization related modules (fp4 quantize/quantize)

November

• [Testing] [API Usability] Minimal example of deploying LLM with flashinfer APIs (e.g. through gpt-fast). #1811
• [Model Optimization] Cosmos Reasoning 7B (details TBD)
• [Testing] Improved unit testing based on escape analysis
• [Testing] Improved integration testing based on escape analysis
• [Model Optimization] MXFP4 gemm perf improvements
• [API Usability] Support FP8-qkv FP8/FP4-output trtllm-gen in FlashInfer prefill/decode wrapper
• [API Usability] Unify qk_scale and o_scale Behavior Between trtllm-gen Attention and flashinfer-jit Attention
• [API Usability] Fused MoE general improvements, including (but not limited to):
  • RFC: Flashinfer MoE Wrappers for vLLM Integration
  • DeepSeek FP8 Block Scaling is not yet implemented in CUTLASS fused_moe #1669
• [API Usability] Attention API consolidation
• [API Usability] Inaccurate API Docstrings for Attention Prefill #1709

[sricketts]

We're working on prioritizing ideas from [Performance]: Custom fused kernel tracking · Issue #25179 · vllm-project/vllm and will add to the roadmap once we have a more concrete plan.

[rainj-me]

Is there any plan for Cute DSL support on following kernels ?

1. RoPE + Set KvCache FP8/FP4
2. Fuse Attention on FP8/FP4
3. General MoE on FP8/FP4
4. Fuse MoE for DS R1, Qwen3 models on FP8/FP4

[sricketts]

Is there any plan for Cute DSL support on following kernels ?

@rainj-me when choosing how to implement kernels in FlashInfer, we consider a number of factors (e.g. perf, maintainability, open vs closed source, etc). Can you some context for your question -- why do you want CuTe DSL implementations of these op patterns (as opposed to some other implementation choice)?

[rainj-me]

Is there any plan for Cute DSL support on following kernels ?

@rainj-me when choosing how to implement kernels in FlashInfer, we consider a number of factors (e.g. perf, maintainability, open vs closed source, etc). Can you some context for your question -- why do you want CuTe DSL implementations of these op patterns (as opposed to some other implementation choice)?

@sricketts , thanks for following up. I'm trying to propose these kernels due to

1. flashinfer already has gemm kernel base on Cute DSL, so it is not introducing a brand new implementations, just to extend the existing functions
2. some existing kernels/solutions only works for specific model and quantization, it is hard to expand to models or quantization

[sricketts]

Is there any plan for Cute DSL support on following kernels ?

@rainj-me when choosing how to implement kernels in FlashInfer, we consider a number of factors (e.g. perf, maintainability, open vs closed source, etc). Can you some context for your question -- why do you want CuTe DSL implementations of these op patterns (as opposed to some other implementation choice)?

@sricketts , thanks for following up. I'm trying to propose these kernels due to

1. flashinfer already has gemm kernel base on Cute DSL, so it is not introducing a brand new implementations, just to extend the existing functions
2. some existing kernels/solutions only works for specific model and quantization, it is hard to expand to models or quantization

Thanks -- let me rephrase to make sure I understand:

• as a general comment you would like to see more open source implementations, so you can extend them for your use cases (e.g. model variations or different quantization recipes).
• since there is already a CuTeDSL gemm kernel in flashinfer, you think that would be a natural choice for the target language for FlashInfer OSS kernels
• I'm guessing you also just like CuTeDSL and think it's a good choice for extensibility :)

Did I get that right?

[samuellees]

@sricketts Could flashinfer provide wheel with AOT kernels? Thanks

[yzh119]

Hi @samuellees instead of calling aot wheel, we call it flashinfer-jit-cache package now, implemented in #1726 (pending some final testing to merge). We also provide another pypi package https://pypi.org/project/flashinfer-cubin/ for cubin kernels.

If only flashinfer-python is installed, everything will be jit compiled.
If user want to have full coverage of pre-built kernels:

pip install -U flashinfer-python flashinfer-cubin
pip install -U flashinfer-jit-cache -i https://flashinfer.ai/whl/cu130 # for cuda 130

[yzh119]

@rainj-me for open-source kernels, cute-dsl will be the default choice for future kernel development (they are still not the default choice for all kernels yet because some bugs/compatibility with earlier architectures).