[Feat][Router] Add TTFT Routing

[chickeyton]

FIX #583

An answer to the feature request TTFT Routing

Dependancies

This PR depends on the to be merged feature FullLookUp in LMCache. Therefore, this PR can not be merged until FullLookUp be merged and be released in an offiical version of LMCache

Ultimate Design

Motivation

The current PrefixAwareRouter and KvawareRouter (in production-stack) route user request purely base on the length the cached prefix. However, it may not be optimal due to the selected instance may have a long pending request queue. A more efficient way is estimating TTFT of the incoming request for each instance, select the one with the least estimated TTFT.

TTFT Estimation

TTFT esitmation is conducted once for each vllm instance on routing a request

ttft = queue_time + cache_transfer_time + compute_time
# where
queue_time  = sum([q.uncached_prefill_tokens for q in pending_requests]) / engine_prefill_tps
cache_transfer_time = sum([transfer_time(c.instance_id, c.location) for c in remote_cached_chunks])
compute_time = uncached_prefill_tokens / request_prefill_tps

queue_time: estimated time of the instance finish all the pending requests, cache transfer times of these requests are neglected
cache_transfer_time: estimated time of transferring matched block cache from other instances to the subject instance, by the number of cached remote chunks and it's size & location type
compute_time: estimated time of the instance process prefill before giving the first token
engine_prefill_tps: realtime measured (per instance) overall token per seconds for the prefill phase in the vllm instance, be measured on the vllm-router side.
request_prefill_tps: realtime measured (per instance) average token per seconds for the prefill phase of each request in the vllm instance, be measured on the vllm instance side
remote_cached_chunks: comes from FullLookUp query to the LMCache registry service
location: either "LocalCPUBackend" or "LocalDiskBackend"
transfer_time(): realtime measured (per instance) cache chunk transfer time depends on the location, be measured on the vllm instance side (by the LMCache code)

Design of this PR

This PR implements only part of the Ultimate Design:

ttft = queue_time + cache_transfer_time
# where
queue_time  = sum([q.uncached_prefill_tokens for q in unfinished_requests]) / engine_prefill_tps
cache_transfer_time = sum([transfer_time(c.location) for c in remote_cached_chunks])

compute_time is eliminated as request_prefill_tps has to be realtime measured inside vllm engine core
transfer_time is hard coded pre-measured times : for "LocalCPUBackend" : 10ms, "LocalDiskBackend" 15ms

Actual Code Changes:

1. Added TtftRouter, like KvawareRouter, it host a centralized LMCache registry service, the vllm instances using the LMCache connector and register/unregister prefix caches from that registry service. On request routing, TtftRouter do a FullLookup (to be merged feature in LMCache) from the registry service to retrieve

2. Added optional commandline argument --tokenizer <model> for specifing the tokenizer path or HF model name, KvawareRouter and TtftRouter will load that tokenizer immediately after starting the LMCache registry service. If not specified they will load the tokenizer with the model specified in the first vllm endpoint on routing the first user request.

3. Added optional commandline argument --lmcache-instances <comman separated list of LMCache instance id> used with --service-discovery static only, if two or more vllm endpoints share a same IP then --lmcache-instances must be specified, otherwise the QueryInstMsg to LMCache queries in KvawareRouter and TtftRouter will give wrong results

4. Added engine_prefill_tps and uncomputed_prefix_tokens measures to RequestStats, uncomputed_prefix_tokens refer to sum([q.uncached_prefill_tokens for q in unfinished_requests])

5. Added optional commandline argument --max-unfinished-queries <num> to muti-round-qa.py to limit the max. no. of unfinished queries for better control of benchmarking/stress tests

Usage:

Assuming two vllm endpoints, vllm router, and Redis service are hosted on the same machine

1. LMCache connector v1 and p2p transfer must be enabled in the vllm engine: https://docs.lmcache.ai/getting_started/quickstart/share_kv_cache.html

prepare 2 LMCache yaml config files: ~/lmcache_config_0.yaml and ~/lmcache_config_1.yaml

example lmcache_config_0.yaml

chunk_size: 256
controller_url: localhost:65432
distributed_url: localhost:8700
enable_controller: true
enable_p2p: true
lmcache_instance_id: instance_0
lmcache_worker_port: 8710
local_cpu: true
local_disk: /tmp/lmcache_0
lookup_url: localhost:8600
max_local_cpu_size: 1.0
max_local_disk_size: 10.0

2. Start Redis server: redis-server --port 8600

3. Start the first vllm endpoint:

LMCACHE_USE_EXPERIMENTAL=True \
LMCACHE_CONFIG_FILE=~/lmcache_config_0.yaml \
CUDA_VISIBLE_DEVICES=0 \
vllm serve Qwen/Qwen3-0.6B --host 0.0.0.0 --port 9081 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'

4. Start the second vllm endpoint like the first one with proper namings, port and CUDA_VISIBLE_DEVICES, LMCACHE_CONFIG_FILE values

5. Start vllm-router with TtftRouter:

vllm-router --port 9091 \
    --service-discovery static \
    --static-backends "http://localhost:9081,http://localhost:9082" \
    --static-models "Qwen/Qwen3-0.6B,Qwen/Qwen3-0.6B" \
    --engine-stats-interval 10 \
    --log-stats \
    --routing-logic ttft \
    --lmcache-controller-port 65432 \
    --lmcache-instances "instance_0,instance_1" \
    --session-key SESSION \
    --tokenizer "Qwen/Qwen3-0.6B"

--routing-logic set to ttft
--lmcache-controller-port must be same as the controller_url's port specified in lmcache_config_*.yaml
--lmcache-instances must be listing the lmcache_instance_id value specified in lmcache_config_*.yaml as two vllm endpoints share the same IP

6. Good to send request to the vllm-router now:

curl http://localhost:9091/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "prompt": "Hello, my name is",
    "max_tokens": 50
  }'

TODO

Todo for implementing the ultimate desgin:

1. Realtime mesure request_prefill_tps in vllm engine and reports via the exsiting Prometheus mechanism, that require at least an extra PR to the vllm repo

2. Realtime mesure transfer_time() in LMCache and reports via the exsiting vllm engine Prometheus mechanism, that require at least an extra PR to the LMCache repo

---

• Make sure the code changes pass the pre-commit checks.
• Sign-off your commit by using -s when doing git commit
• Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

Detailed Checklist (Click to Expand)

Thank you for your contribution to production-stack! Before submitting the pull request, please ensure the PR meets the following criteria. This helps us maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Please try to classify PRs for easy understanding of the type of changes. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Feat] for new features in the cluster (e.g., autoscaling, disaggregated prefill, etc.).
• [Router] for changes to the vllm_router (e.g., routing algorithm, router observability, etc.).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• Pass all linter checks. Please use pre-commit to format your code. See README.md for installation.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Please include sufficient tests to ensure the change is stay correct and robust. This includes both unit tests and integration tests.

DCO and Signed-off-by

When contributing changes to this project, you must agree to the DCO. Commits must include a Signed-off-by: header which certifies agreement with the terms of the DCO.

Using -s with git commit will automatically add this header.

What to Expect for the Reviews

We aim to address all PRs in a timely manner. If no one reviews your PR within 5 days, please @-mention one of YuhanLiu11
, Shaoting-Feng or ApostaC.

[gemini-code-assist]

Summary of Changes

Hello @chickeyton, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates a novel Time To First Token (TTFT) routing mechanism into the vLLM router. This enhancement aims to optimize request distribution across available backends by predicting the fastest path to the first generated token, thereby improving overall system responsiveness and efficiency. The changes also include necessary updates to the request monitoring and benchmarking tools to support and evaluate this new routing approach.

Highlights

• New TTFT Routing Logic: Introduced a new Time To First Token (TTFT) routing strategy that intelligently selects the optimal backend by estimating the time it will take to generate the first token, considering KV cache hits and current engine load.
• Enhanced Prompt Handling: Implemented a robust utility to extract prompts from various request formats (completion, chat, multimodal) for accurate tokenization, which is essential for effective KV cache lookups in the new routing logic.
• Improved Request Statistics: Extended the request statistics monitoring to track critical metrics such as 'engine_prefill_tps' (tokens per second during prefill) and 'uncomputed_prefix_tokens', providing the necessary data for the TTFT routing algorithm's estimations.
• Benchmarking Tool Updates: Modified the multi-round QA benchmark to include a 'max_unfinished_queries' parameter, allowing for better control over concurrent requests and more accurate simulation of load during performance testing of the new routing strategy.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new TTFT (Time To First Token) routing strategy, which is a significant feature. The implementation spans routing logic, statistics collection, and request processing, and includes updates to the benchmarking tool. The overall approach is sound, but I've identified a critical bug in the new TtftRouter where an exception is created but not raised. I've also left several comments to improve code clarity, maintainability, and robustness, such as renaming misleading variables, removing magic numbers, and improving an algorithm's efficiency. Addressing these points will strengthen the new routing logic.

[chickeyton]

This PR depends on to be merged FullLookup feature of LMCache, so cannot be merged yet, but please take a look of the design if u have time, thanks!

@KuntaiDu @ApostaC