A step-level simulator measuring the interaction between chunked prefill and KV cache pressure in continuous batching LLM serving.
For full methodology and design decisions see design.md.
Chunked prefill divides long prompts into smaller chunks to interleave with decode steps, improving fairness. But each chunk incrementally consumes KV cache slots. When a new request arrives between chunks and exhausts the remaining KV budget, the in-progress prefill is interrupted — creating a failure mode that upfront admission control cannot prevent.
no_chunk Full KV reserved upfront. Reject if insufficient. reserve Full KV reserved upfront. Prefill executed in chunks. block Incremental KV allocation. Block and retry if a chunk fails. steal Incremental KV allocation. Preempt decode to free space.
Mixed workload (60% short, 40% long), KV budget=24MB:
Policy tput clean reject block preempt chunk_waste TTFT_p99 no_chunk 2.67 0.599 0.401 0.000 0.000 0.000 8ms reserve 2.67 0.599 0.401 0.000 0.000 0.000 8ms block 2.58 0.581 0.035 1.539 0.000 0.896 8ms steal 1.52 0.389 0.013 0.396 0.201 0.731 8ms
No_chunk is strictly better than block. Same effective throughput. Zero GPU waste vs 89.6% chunk waste. Upfront rejection wastes no compute. Mid-prefill blocking does.
Chunk waste rate is the critical metric. Standard metrics (throughput, rejection rate) hide partial prefill waste. chunk_waste_rate captures GPU compute discarded on blocked prefills. 89.6% waste means 9 of every 10 prefill chunks are discarded.
Retry amplifies chunk waste without recovering throughput. block_rate > 1.0 means requests retry multiple times. Each retry discards all prior chunks and starts over. Retry in block policy is net-negative vs upfront rejection.
Steal is catastrophic. 73.1% chunk waste combined with 20.1% preemption. Creates a destructive cycle: steal decode, block anyway, retry, steal again.
The practical implication: Chunked prefill under KV pressure requires either full KV reservation before starting any chunk (= no_chunk behavior) or conservative admission based on projected total footprint. Naive incremental allocation wastes significant GPU compute with no throughput benefit.
chunked-prefill-kv-sim/ ├── src/ │ ├── init.py │ ├── config.py │ ├── workload.py │ ├── simulator.py │ └── analysis.py ├── results/ ├── plots/ ├── LICENSE ├── design.md ├── README.md ├── requirements.txt └── run.py
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python run.py
Outputs: results/results.csv results/summary.txt plots/throughput_vs_arrival.png plots/waste.png
Closes a three-part loop:
prefill-chunking-profiler: chunked prefill improves fairness kv-cache-aware-scheduler: KV footprint as admission signal chunked-prefill-kv-sim: what happens when these interact under pressure
MIT License. See LICENSE for details.
Joao Felipe De Souza 2026