[Flow Control] Inter-flow policies to support batch inference and to prevent starvation of flows

[lioraron]

What would you like to be added:

1. Enable to support the following requirement for batch inference via an inter-flow policy: When there is a high load of interactive workload - forward mostly interactive requests with a trickle of batch requests. When the interactive workload is low - forward a growing stream of batch requests.
   While the above is considering 2 flows (interactive and batch) - the policy could be generic, supporting any number of flows with a hierarchy, minimum forwarding quantity per flow (the minimum quantity can correlate to the relative priority of the flow), and a logic to calculate the forwarding quantities per flow based on flows' load metrics.

2. In the current implementation the default inter-flow policy has the risk of starvation of lower priority flows. The requirement here is to implement a policy that prevents starvation (e.g. the suggestion above), and set this policy as the default policy.

Why is this needed:

1. To facilitate batch inference and additional use cases.
2. To prevent starvation of lower priority flows.

[LukeAVanDrie]

Each "flow" in our system is represented as a composite key:

type FlowKey struct {
  ID: string // logical grouping ID (tenant name, client, workload, model, etc)
  Priority: int // priority *relative to other flows* (not per-request priority)
}

(1) We first prioritize across the FlowKey.Priority axis. This is currently not an extension point; we serve flows in strict priority order descending.

(2) Once the priority band is selected, we have a set of FlowKeys all with priority p. This is where the InterFlow policy takes effect. It prioritizes across FlowKey.ID axis, enforcing fairness within a priority band.

(3) Unrelated to your issue, but for completeness... once a FlowKey is selected by applying (1) then (2), we select the HoL request via an IntraFlow policy which defines the ordering of the respective FlowKey's queue.

---

Can you confirm my understanding that you are looking to make (1) extensible? This would be the layer for adding WFQ, DRR, etc to ensure every priority band gets a minimum quantum of service.