Extended E2E Tests with More Scenarios

[dumb0002]

Extend the E2E tests for scalefromzero to cover more scenarios (e.g., multiple deployments for the same base model, multiple EPPs, multiple inferencepool version, etc)

[shuynh2017]

@dumb0002, is there something that I can help here? I ran the scalefromzero cases just yesterday.

[dumb0002]

@shuynh2017, yes! Could you start by working on a PR to extend these tests to cover the scenario of having multiple EPPs for the same inferencepool version? thx!

[shuynh2017]

@dumb0002, thanks. Newbie question: can this be done in a Kind cluster? Any pointers would be great.

[dumb0002]

@dumb0002, thanks. Newbie question: can this be done in a Kind cluster? Any pointers would be great.

@shuynh2017, yes! You can run this test in a local kind cluster indeed.

For this scenario we need the following (summary):
a) Deploy a second EPPs (with respective inferencePool, model serving deployment but with zero replicas) in the same namespace where llm-d infra is deployed;
b) Create a second VA object associated with the model serving deployment created above;
c) Send inference requests to both EPPs (old and new on) to trigger parallel scaleFromZero events;

Please, let me know if you have any further questions.

[shuynh2017]

@dumb0002, I used MULTI_MODEL_TESTING option with kind installation, it created 2 inference pools and 2 set of model pods but HTTPRoute only references 1 inference pool, and there's only 1 EPP - not as expected in https://gateway-api-inference-extension.sigs.k8s.io/guides/serving-multiple-inference-pools-latest/. Will have to fix this first?

[dumb0002]

@dumb0002, I used MULTI_MODEL_TESTING option with kind installation, it created 2 inference pools and 2 set of model pods but HTTPRoute only references 1 inference pool, and there's only 1 EPP - not as expected in https://gateway-api-inference-extension.sigs.k8s.io/guides/serving-multiple-inference-pools-latest/. Will have to fix this first?

@shuynh2017, for this test we would need to extend it to also deploy a second EPP with the respective HTTP Route for it created.

Some clarification questions: a) what issue are you referring to above? and b) which component in your environment, did you set/configure the var MULTI_MODEL_TESTING ?