Fixed epp pod starting but not working when using multiple schedulingProfiles

[learner0810]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fixed epp pod starting but not working when using multiple schedulingProfiles

Which issue(s) this PR fixes:

Fixes # #1697

Does this PR introduce a user-facing change?:

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	f441bfd
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/68e8af45fe08ef0007c6bd63
😎 Deploy Preview	https://deploy-preview-1698--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: learner0810
Once this PR has been reviewed and has the lgtm label, please assign nirrozenbaum for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @learner0810. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[learner0810]

ping @ahg-g

[nirrozenbaum]

@learner0810 not sure if we want to test that on startup. we had that discussion in #1169, where I was actually suggesting to check it but was convinced by others that we shouldn’t.

there might be plugins combinations that work only if they are set together (e.g., in llm-d, pd profile handler will work correctly only if p filter is defined in prefill scheduling profile and d filter in decode profile). I think we don’t want to validate these kind of things cause it has no end.
it should probably be the in the scope of a sanity test to catch those issues.

[learner0810]

@learner0810 not sure if we want to test that on startup. we had that discussion in #1169, where I was actually suggesting to check it but was convinced by others that we shouldn’t.

there might be plugins combinations that work only if they are set together (e.g., in llm-d, pd profile handler will work correctly only if p filter is defined in prefill scheduling profile and d filter in decode profile). I think we don’t want to validate these kind of things cause it has no end. it should probably be the in the scope of a sanity test to catch those issues.

Even if misconfigured plugins can successfully route requests (even if routed to the wrong Pod), this may be tolerable. However, if misconfiguration causes requests to fail, we must test this during startup.

Here is my understanding . Please correct me if I'm wrong.

[kfswain]

I think I'm okay with flagging this edge case. Esp because it references the default handler. Making our framework easier to use is a big plus wrt adoption.

/ok-to-test

[kfswain]

I think we don’t want to validate these kind of things cause it has no end.

This is a fair point.... but the profile handler is kind of nuanced and a more complex part of our system. It is probably a real gotcha to try and do something sophisticated (multiple profiles) and then cut yourself on the no handler implementation.

[nirrozenbaum]

I think we don’t want to validate these kind of things cause it has no end.

This is a fair point.... but the profile handler is kind of nuanced and a more complex part of our system. It is probably a real gotcha to try and do something sophisticated (multiple profiles) and then cut yourself on the no handler implementation.

I’m fine with that.
we just need to cut the validation somewhere.
specifically if one configures handler incorrectly the logs show a very clear message about that.
so from one hand it’s easy the identify, from the other hand it’s correct that we can fail on bootstrapping.

for me it feels like testing the internals of plugins, but if you feel we want it on startup let’s go for it.