[Flow Control] Reconciliation of the flow control logic with the scheduling logic

[lioraron]

What would you like to be added:
In the current implementation it is possible that the decision in the flow control component conflicts with the subsequent decision in the scheduling component. For example: flow control decides (based on logic 1) that overall there is some capacity in the pool and forwards some requests - hoping these will land in model replicas that have capacity, but then the scheduler decides (based on logic 2) to route these requests to a more busy model replica where the requests end up being queued locally. This can cause underutilization of resources and excess latency of inference requests.
The requirement is to add a mechanism to enable better knowledge sharing between these components, to ensure that their decisions align.

Why is this needed:
Without the above - conflicts in the decisions of flow control and the scheduler can conflict, causing underutilization of resources and excess latency of inference requests.

[LukeAVanDrie]

/assign

I am working on a proof of concept for this now. It does three things:

1. Filters "saturated" pods unless every pod is saturated in which case we fail open and let the other scheduling plugins choose the best option.
2. Below pool saturation, scores pods s.t. we bin pack on the most loaded (but not yet saturated) pods maximizing hardware efficiency
3. At and beyond pool saturation, scores pods s.t. we spread traffic to the least loaded pods maximizing latency.

The operational regime switching between (2) and (3) are also not a sharp cutoff. It smoothly transitions between these two strategies.

(1) strictly protects the pool. (2) and (3) give the scheduler a lot of flexibility to make its own optimization decisions but feeds forward sufficient pool and pod saturation context to weight in its options.

IMO, (1) is the only critical requirement here. (2) and (3) are nice to haves that smartly handle different scheduling objectives based on the service load.