Standalone EPP - Endpoint Discovery Using Pod Selector Investigation

[capri-xiyue]

What would you like to be added:
EPP As a Standalone Request Scheduler mentioned using Endpoint Discovery with Pod Selector . This issue is to track the investigation of endpoint discovery with pod selector.

Why is this needed:
The endpoint picker (EPP), an intelligent scheduler for LLM inference requests, is currently deployed as an ext-proc service tightly coupled to the Kubernetes Gateway API, a setup that adds unnecessary operational overhead for non-serving workloads like Reinforcement Learning (RL) post-training. The proof of concept for a "Standalone EPP" aims to solve this by creating a decoupled deployment mode. This new approach will be tested by packaging the EPP and an Envoy proxy as a single sidecar-based unit, bypassing the Gateway API and instead using a simple command-line pod selector to discover and manage model server endpoints, thereby lowering the barrier for adopting the EPP's advanced scheduling logic for critical RL and batch inference jobs.

[capri-xiyue]

/assign @capri-xiyue

[nirrozenbaum]

@ahg-g I’m not sure I understand the motivation behind this issue and would be happy to understand.
this issue is not about bypassing Gateway API, it’s about bypassing InferencePool CRD which is one of GIE main building blocks.

specifying pod labels selector as cmd-line is against k8s principles and is not matching how GIE works today where the selection label can changed dynamically in runtime.
how do you handle conflicts if one species the cmd-line arg and uses the crd?
(not even mentioning the multi port that was ignored).

why do we need this? standalone epp without ext proc can work with InferencePool CRD

[capri-xiyue]

Re - "For conflicts if one species the cmd-line arg and uses the crd"
The EPP will have validation that either use inferencepool or use pod label selector as cmd-line. Users can't specify both.
When users specifies the pod label selector via cmd-line agrs and create a inferencepool, as such inferencepool is not specified in EPP, I think users won't hit any issues. Just an unused inferencepool resources lives in the cluster?

[ahg-g]

@nirrozenbaum I tried to capture the feature request in #1838; do you mind moving the discussion there since the answer to your question requires alignment on a higher level proposed strategy.

[capri-xiyue]

The POC of the feature has been merged in #1833 (I've manually tested it works e2e). To avoid regression failure, I will add UT and e2e tests later. But before the UT and e2e tests in place, the feature may encounter regression failure.