EPP HA deployment

[liu-cong]

What would you like to be added:

Our current user guide sets 1 replica for EPP:

gateway-api-inference-extension/config/manifests/inferencepool-resources.yaml

Line 39 in 92431f5

replicas: 1

We should provide the option to run EPP in HA mode. I suggest the following priorities:

• Short-term: Active-passive mode. This is a good starting point, as in some benchmarks we saw EPP can handle >500 QPS, and 10s (potentially 100s) of model servers, without a clear bottleneck. So a single leader EPP is sufficient for many use cases. Currently we run EPP without leader election code, we will need to modify EPP to support leader election, and run some test to verify the downtime if the leader dies.

• Mid-Long term: Support active-active EPPs, with options to shard the InferenceModels and InferencePool.
  With multiple active EPP replicas, we expect decreased performance on stateful operations such as queuing and prefix cache aware routing. This will require some experiments to quantify the impact.

If there are too many inference models, or if if the pool becomes very large, we may need to further shard them.

Why is this needed:

[nirrozenbaum]

for the mid-long term, please keep in mind that multiple models are running on a single pod, and there might be a setup where sharding is not possible.
for example -
pod1 runs adapter1,2,3
pod2 runs adapter2,3,4
pod3 runs adapter 3.4,1
pod4 runs adapter 4,1,2

in this setup if epp is scraping from one pod it should scrape from all of them to get the full state.
If sharding was possible that is the ideal solution but unfortunately I think this is not going to work.

I also thought about this topic and came into conclusion that we need a high speed distributed cache, something like redis.
see this issue - #671

[kfswain]

for the mid-long term, please keep in mind that multiple models are running on a single pod, and there might be a setup where sharding is not possible. for example - pod1 runs adapter1,2,3 pod2 runs adapter2,3,4 pod3 runs adapter 3.4,1 pod4 runs adapter 4,1,2

in this setup if epp is scraping from one pod it should scrape from all of them to get the full state. If sharding was possible that is the ideal solution but unfortunately I think this is not going to work.

I also thought about this topic and came into conclusion that we need a high speed distributed cache, something like redis. see this issue - #671

I think we would actually need to benchmark/load test this to be certain. Adding redis adds latency to a decision we are already trying to make essentially real time. Redis, while fast, would still be slower than in mem.

Agreed that sharding is imperfect, but I think we actually would need to test this before we can be sure.

[liu-cong]

for the mid-long term, please keep in mind that multiple models are running on a single pod, and there might be a setup where sharding is not possible.

At that point we may need a control plane component to place/shard those adapters to pods, and align that sharding decision with EPPs. We should gather some real use case before going too far there though.

We haven't benchmarked but I think EPPs having the global view of all pods will likely be enough for a long time.

[rlakhtakia]

/assign

[danehans]

@rlakhtakia thanks for taking on this issue. When you have a moment, do you mind providing an update on your progress?