Adjust request "processing time" to current load

[mayabar]

Current state:
Request processing time is composed of the prefill and decode parts.
If prefill is executed on the same instance, its time can be defined by one of the following options:

• time-to-first-token (TTFT) parameter: represents prefill time independent of prompt size
• prefill-overhead + prefill-time-per-token (PTPT) * number-of-propmpt-tokens: prefill time which takes into account the prompt length
  In case of disaggregated prefill (prefill executed on a different pod), the time is calculated as kv-cache-transfer-latency * number-of-propmpt-tokens

Decode time is calculated as:
inter-token-latency * number-of-propmpt-tokens

What would you like to be added:
Allow support to adjust all the above timeouts based on the current load (i.e., the number of requests currently running in parallel)
Introduce a new configuration parameter load-factor which defines the multiplier for all time delays when the load is at maximum.
All time configuration values should be calculated using linear interpolation between the number of requests [1-max-num-seq] and the time range [time-defined-in-config, time-defined-in-config*load-factor]

The formula:
time = time_defined_in_config * [1 + (load_factor - 1) * (current_requests_num - 1) / (max_num_seq - 1)]

Why is this needed:
The real vLLM's performance depends on the current load. The simulator should mimic this behavior for realistic results.

[pancak3]

Hi Maya, can I take this

[mayabar]

@pancak3 sure ;)

[pancak3]

The formula:
time = time_defined_in_config * [1 + (load_factor - 1) * (current_requests_num - 1) / (max_num_seq - 1)]

Formatted:

$$ process\_time = time\_defined\_in\_config \times \left[1 + \frac{(load\_factor - 1) \times (current\_requests\_num - 1)}{(max\_num\_seq - 1)}\right] $$

---

Hi @mayabar, could you please help me understand the following? Thanks!

1. MaxNumSeqs is not validated (see below func); should it always be >= 1 or >= 0? This affects whether to check zero division when impl

llm-d-inference-sim/pkg/common/config.go

Line 291 in 08d4613

func (c *Configuration) validate() error {

2. Introduce a new configuration parameter load-factor which defines the multiplier for all time delays when the load is at maximum.

(current_requests_num - 1) This part is straightforward, which only applies when there are parallel requests, the extra delays are effective.
But the new param, called *load*-factor, is confusing. Seeing(load_factor - 1), it means load_factor is greater or equal than 1, otherwise the formula decreases the process_time. Perhaps the naming is confusing or the formula can be adjsuted, could you please help?

3. Once the formula is cleared, can I encapsulate the following in a function getProcessTime( ...) and return the adjusted totalMillisToWait?
   

   llm-d-inference-sim/pkg/llm-d-inference-sim/simulator.go

   Line 651 in 08d4613

   totalMillisToWait := s.getTimeToFirstToken(nPromptTokens, doRemotePrefill) + s.getTotalInterTokenLatency(nGenTokens)

[mayabar]

1. you are right, the validation check is missing, can you please add validation that the value is positive (>0). the formula should be used for the MaxNumSeqs >= 2, for MaxNumSeqs=1 - we should just use base time and ignore the load_factor
2. you right, the name 'load-factor' a little bit confusing, we came with the updated name 'time-factor-under-load' ;) In comment you can say that this is a multiplier for all local processing times on the maximum load, other values are distributes linearly. Pay attention that transfer time should not be affected by this multiplier, also standard deviation values are not affected too.
3. Unfortunately, the change cannot be done in one place - getProcessTime. The adjustment should be done in all places the time are used (maybe it worth to add wrapping function for all values). In case of "not-streaming" request the total processing time contains sometimes the kv-cache-transfer value which is not affected by the local load. In case of streaming request, total time is not calculated.
   Hope this will help, please let me know if you have additional question/doubts/comments, thanks

[mayabar]

Additional important note: the formula above has a problem, the fixed version:

$$ process\_time = time\_defined\_in\_config \times \left[1 + \frac{(load\_factor - 1) \times current\_requests\_num}{(max\_num\_seq - 1)}\right] $$

current_requests_num is number of currently running requests, which valid value is [0, MaxNumSeqs-1]