[RFC] Automatic Scorer-Plugin Weighting

[vMaroon]

Problem Statement

The llm-d inference scheduler relies on multiple scoring plugins to select a target pod for each inference request. These scorers generally fall into two conceptual families:

• Distributive scorers

  • Aim to evenly distribute load across pods
  • Reduce hotspotting and compute underutilization
  • Commonly based on queue depth, in-flight requests, or utilization signals

• Sticky scorers

  • Aim to preserve execution locality
  • Increase scheduling stickiness to reduce unnecessary request movement
  • Often leverage pod-level affinity or reuse signals

The scheduler must balance these two families to achieve optimal throughput and latency.

Today, this balance is controlled through statically configured scorer weights. However:

• Static weights are difficult to tune due to differing scorer variances despite normalization
• Optimal weighting depends on runtime cluster conditions
• Static configuration does not adapt to heterogeneous or time-varying workloads

Proposal

Introduce automatic scorer plugin weighting as an optional scheduler capability.

When enabled, the scheduler dynamically adjusts the relative influence of distributive and sticky scorer families at runtime based on observed cluster load conditions.

Automatic Weighting Behavior

• Scorers are grouped into distributive and sticky families
• The scheduler observes load imbalance signals, such as:
  • Queue depth variance
  • In-flight request skew
  • Utilization imbalance
• When load imbalance increases:
  • Distributive scorers are up-weighted to spread traffic
• When load is balanced:
  • Weighting converges toward an equilibrium between the two families
• Adjustments are gradual and bounded to prevent oscillation

Configuration

• A new configuration field enables or disables automatic weighting
• When enabled:
  • An equal aggregation of the two families sets the starting weights
    • E.g., a configuration with prefix-cache-scorer (sticky), queue-scorer (distributive) and kv-cache-utilization-scorer (distributive) would be 2:1:1.
  • Runtime adjustments are applied relative to these baselines
• Optional: users may bias the equilibrium point by choosing non-equal baseline weights (practically setting the user-configured weights as the baseline)

[nirrozenbaum]

duplication of kubernetes-sigs/gateway-api-inference-extension#1992?

[vMaroon]

I see a duplication of the motivation but i'm not seeing the proposal - is one going to be public?

[kfswain]

I see a duplication of the motivation but i'm not seeing the proposal - is one going to be public?

Yes

[ahg-g]

I think a first step here is to come up with a benchmark to show the limitation of the existing approach. @kaushikmitr I am interested to see how the latency predictor behave in this case.

[kaushikmitr]

this is interesting, can you shed light on this "Runtime adjustments are applied relative to these baselines". How are you thinking of applying the scorer adjustments during runtime?

[kaushikmitr]

I think a first step here is to come up with a benchmark to show the limitation of the existing approach. @kaushikmitr I am interested to see how the latency predictor behave in this case.

The limitation of the existing approach is that the plugin weights depend on workload and its hard to configure. I have some benchmarks on this here
Latency predictor scorer has no weights (at least not yet), and in absence of slo it just sends requests to lowest predicted latencies. But if we can tune the weights real time to achieve optimal performance, that will be great!

[vMaroon]

Holding issue as Nir and Kellen are already working on this problem.

/hold