Batch-invariant chunked prefill with FlashInfer Backend

[rajagond]

Hi,

I would like to better understand batch-invariant chunked prefill when using the FlashInfer backend. The SGLang blog mentions that "the chunking logic aligns the truncation point with an integer multiple of split_kv_size." Could you explain why the truncation point must align with a multiple of split_kv_size, and why the choice of truncation point is important in this context? Additionally, would it be sufficient to disable KV splitting and use BatchedKVCachePaged to avoid this constraint?

More broadly, would fixing query_length to a particular value, for example 100, be sufficient? If not, why?

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• FA3 Paged attention is slower than FA2 in SGLang #1340
• [Help needed] How to jointly compute prefill and decode phases in flashinfer? #1030
• Why is there no .whl installation version for 0.2.6? #1269
• [QST] B200 FP8 Prefill Kernels #1411
• How to handle ADDITIONAL_FUNC_PARAMS using C++ API loaded from .so file? #2003
• Low performance of POD Attention compared to BatchPrefillWithPagedKVCache #1022
• TRTLLM-gen decode kernels do not support variable length queries #1832
• [Bug] VariableBlockSparseAttention fails when num_kv_head * num_blocks_per_row > 32768 #1367
• attention kernel: fp8 kv is 30% slower than bf16 kv #1753

👤 Suggested Assignees

• Edenzzzz
• tp-nan
• CSEEduanyu
• yigithanyigit
• XuZhang99

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!