Adding support for SLO-driven model-based scaling

[atantawi]

Current autoscaler considers the utilization of the KV cache and the request queue length as indicators of saturation, and it accordingly scales up or down the number of replicas, without consideration of user-perceived performance metrics such as TTFT and ITL, nor potential specifications of targets (SLOs) for such metrics.

This issue addresses the need for an SLO-driven, model-based autoscaling. The reason for a performance model is to assess the number of replicas required to attain a given SLOs, based on the observer/predicted load on the servers.

Several tasks are envisioned to address this issue.

• Add some some basic functionality.

• Add a queueing model to analyze and size a replica of an inference server. The model may be called to (1) analyze the steady state (server state over a period of time, such as a control cycle, as opposed to a given current state) and produce TTFT and ITL values for a given load (request rate and input/output token characteristics) and (2) size a server by providing the maximum request rate for a given load in order to attain TTFT and ITL SLOs. The queueing model may have a handful of internal parameters, which may be obtained off-line or on-line, preferably the latter. (PR Enhance queueing model used by queue analyzer #727 )
• Add an on-line model tuner to continuously estimate the queueing model internal parameters using observations and standard filtering techniques. (PR Basic model tuner functionality #743 )

• Conduct experiments to validate queueing model and on-line model tuner.

• Add means to collect metrics from the Scheduler related to the offered request rate, as opposed to the successfully completed requests. The former should be used to size the server, whereas the latter should be used to tune/calibrate the queueing model.

• Add mechanism to specify SLOs. Two alternatives are possible: (1) explicit by modifying the APIs to allow user/admin to specify absolute/relative values for TTFT and ITL SLO targets, and (2) implicit by having WVA automatically setting target values based on observations and optimizing an internal objective.

• Optionally, add a predictor of load (request rate and token distributions) for the next control cycle, given the statistical history, potentially using time-series techniques.

• Add a new model-based scaling engine which employs above queueing model and on-line model tuner.

Notes:

• TTFT measure should be expressed as a relative measure to the number of prompt tokens in the requests, e.g. TTFT per 100 tokens.
• Queueing model need not be exact, as it is used solely to estimate maximum load to attain a given SLOs, as opposed to performance evaluation of performance metrics.

[atantawi]

Supersedes issue #70

[James-QiuHaoran]

Hi @atantawi, I would like to suggest an SLO-aware approach.

This logic has been effectively used in recent research such as DynamoLLM (HPCA '24) and ModServe (SoCC '25).

This approach uses Maximum Sustainable Throughput under SLO derived from profiling, and the logic is simple:

1. For a given model and hardware configuration, profile to find the maximum token throughput ($T_{max}$) that stays within the TTFT (Time to First Token) and TBT (Time Between Tokens) SLO thresholds.

   • Example: If an instance type can handle $800$ tokens/s before the 95th percentile TTFT exceeds the limit, then $T_{max} = 800$.

2. Monitor the real-time aggregate demand ($D_{total}$) in tokens/s across the cluster.

3. The number of required replicas ($N$) is calculated as:

   $$N = \left\lceil \frac{D_{total}}{T_{max} \cdot (1 - \text{buffer})} \right\rceil$$

A safety buffer (e.g., $0.15$) is applied to account for overhead and traffic bursts.

If possible, we can help work on adding SLO-awareness feature to llm-d.

[atantawi]

Hi @James-QiuHaoran , Good to hear from you and nice suggestion.
Yes, that is a viable way, however the issue with that approach, in my opinion, is the need for off-line profiling. Profiling has to be performed for a very large space: model, GPU, input/output tokens, and SLO values. Not to mention configuration variations (and version) of vllm servers. Correct?

[James-QiuHaoran]

That's a valid concern -- precise profiling would indeed be a bottleneck for new vLLM versions.

However, we can avoid explicit offline profiling by leveraging curve fitting with existing data, i.e., we fit a throughput-latency curve (e.g., tok/s vs. TTFT) using historical telemetry from the deployed vLLM instance. This naturally accounts for the specific software version, GPU, and workload mix.

Then given the desired SLO (e.g., 1000ms TTFT), we simply look up the corresponding maximum throughput on the fitted curve to determine per-instance capacity.

This allows the scaling logic to stay SLO-aware and self-calibrating without the manual overhead of a high-dimensional profiling matrix. What do you think?