Scale testing for the EPP

[kfswain]

With the llm-d sim server available, high scale testing of Inference Gateway is possible. Before GA, it would be good to know some upper limits of the default EPP.

Dimensions of test:

• prompt length
• raw QPS
• a sample of the Pareto frontier of those two dimensions

Note: Also record the Resources Limits configured for the EPP container

[Grayson-LaFleur-Google]

Ack. Working on getting these results now.

[nirrozenbaum]

Hi @Grayson-LaFleur-Google,
since first RC is planned in less than a week, I’m pinging issue owner for status update.

where do we stand with the scale testing and what is the completion time estimation?

[kfswain]

I can answer this, we are very close.

This should be done in the next 1-2 days. I deployed an EPP for Grayson that gave innaccurate results so we had to throw out some testing. But we still have solid baselines & have fixed some issues that popped up from scale testing, so basically unblocked and collecting results is all that's left.

[nirrozenbaum]

what is the result we expect from this issue? what is the definition of done?
is it - documenting how to run EPP with HA? changing all manifests to work in HA? both?

[kfswain]

Results will be:

• Having a solid knowledge of our traffic limitations for a given set of hardware.
  • Ex: for 32Gi mem & 16 CPU cores; I know the EPP can handle X prompt size at Y QPS. Specifically set Y to the descrete values of 10k,5k, & 1k.

DOD is the results of this benchmarking to help with capacity planning. Follow ups would be to provide external benchmark tools so that a user could run the same test themselves and see what sort of performance they can expect (variability will be somewhat high given a user can define arbitrary scheduling plugins)

[kfswain]

This is related to HA in that we are planning on an active-passive model, so this gives us the scale limits of the single active member. Actual HA deployment is a seperate issue see: #1337