Enable EPP to support endpoint discovery using pod selector #1833

capri-xiyue · 2025-11-07T19:47:40Z

What type of PR is this?

/kind feature

What this PR does / why we need it:
See #1779
Which issue(s) this PR fixes:

Part of #1779

Does this PR introduce a user-facing change?:

NONE to existing features with inferencepool. 
But users can use epp without inference pool via args 
`
        - --endpoint-selector
        - "app=vllm-llama3-8b-instruct"
        - --endpoint-target-ports
        - "8000"
`

Signed-off-by: Xiyue Yu <[email protected]>

netlify · 2025-11-07T19:47:47Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`11a8a68`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/691f6d4f816c030008a428e0
😎 Deploy Preview	https://deploy-preview-1833--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

capri-xiyue · 2025-11-07T19:54:04Z

No need to review it right now. I just made the CUJ of standalone epp work without inferencepool. Still need to fix the e2e and ut

elevran

cursory review to provide initial feedback (realizing this is work in progress)
The main question I have (and might be worth mentioning in the PR description) is the need for a new abstraction/type (EndPointsPool). A naive/simple solution (which perhaps does not work...) would be to copy the selector and port array into a Go InferencePool object and use datastore.PoolSet() along with disabling the Pool notification/reconciliation so it does not overwrite with nil.
Hopefully the rest of the code should not care or depend on the pool's origin (from command line or the API server)

cmd/epp/runner/runner.go

pkg/epp/controller/pod_reconciler.go

pkg/epp/controller/inferenceobjective_reconciler_test.go

capri-xiyue · 2025-11-14T23:39:02Z

assign @ahg-g for early review.

capri-xiyue · 2025-11-14T23:39:19Z

assign @kfswain for early review

cmd/epp/runner/runner.go

pkg/epp/controller/pod_reconciler.go

pkg/epp/datalayer/endpointsPools.go

pkg/epp/datastore/datastore.go

Signed-off-by: Xiyue Yu <[email protected]>

pkg/epp/datalayer/endpoint_pool.go

pkg/epp/server/controller_manager.go

pkg/epp/datastore/datastore.go

cmd/epp/runner/runner.go

pkg/epp/datalayer/endpoint_pool.go

elevran · 2025-11-19T07:13:52Z

My general point is that having a canonical internal type that represents the pool is useful, and abstracts how the EPP was configured with the pool.

Do we actually need pool as a canonical type or concept?
If so, and given the current code base and future changes in #1838 - is it warranted to replace the current InferencePool Go with a new type?

Given this from #1838:

Orchestrator agnostic deployment mode: static configuration of endpoints with zero dependency on k8s as a control plane, which means the reconcilers are not started as well.

k8s-native deployment modes
2.1) No dependency on Gateway API or extension CRDs, endpoints are discovered using a pod selector and port flags
2.2) No dependency Gateway APIs, extension CRDs are used to enable a richer feature set for EPP, but the proxy is statically
...

The Inference Pool is not needed at all in (1) and all reconcilers should be disabled, option (2.1) works well with a "fake" pool that has the minimal attributes set and a no pool reconciler, and (2.2) or later use the full CRD.
I don't think the internal type improves or changes any of that.

Ultimately we deal with inference endpoints (BTW: agree with @kfswain comment that the term is not used consistently across and we should make it more explicit). The InferencePool is merely one way to define attributes of service discovery for relevant endpoints.

ahg-g · 2025-11-19T16:33:41Z

I think we are going back and forth on a non material issue; both paths are reasonable and one can't argue that the one in this PR is wrong or harmful; it is also an implementation detail that I recommend to empower contributors to make.

pkg/epp/datalayer/endpoint_pool.go

cmd/epp/runner/runner.go

pkg/epp/requestcontrol/director.go

test/integration/epp/hermetic_test.go

pkg/epp/datalayer/endpoint_pool.go

cmd/epp/runner/runner.go

pkg/epp/datalayer/endpoint_pool.go

pkg/epp/server/runserver.go

ahg-g · 2025-11-20T19:30:51Z

/approve

k8s-ci-robot · 2025-11-20T19:31:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, capri-xiyue

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ahg-g · 2025-11-20T22:00:57Z

/retest

ahg-g · 2025-11-20T22:37:00Z

/lgtm

capri-xiyue added 7 commits October 31, 2025 16:18

partial draft

5945cfb

refactor

370f1a3

Signed-off-by: Xiyue Yu <[email protected]>

fixed some ut

85e4622

make epp controller ut pass

579b17e

make ut pass

e9d704d

fixed build

42cb218

fixed build

943c53d

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Nov 7, 2025

k8s-ci-robot requested a review from elevran November 7, 2025 19:47

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 7, 2025

k8s-ci-robot requested a review from nirrozenbaum November 7, 2025 19:47

capri-xiyue changed the title ~~Enable EPP to support endpoint discovery using pod selector~~ [WIP] Enable EPP to support endpoint discovery using pod selector Nov 7, 2025

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 7, 2025

capri-xiyue added 3 commits November 10, 2025 12:54

fixed build failure

1e38821

fixed lint

30fd667

fix format

9e85377

elevran mentioned this pull request Nov 11, 2025

Standalone EPP #1838

Open

elevran reviewed Nov 13, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 14, 2025

merge conflicts

43c87fa

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 14, 2025

fixed import format

0267569

ahg-g reviewed Nov 14, 2025

View reviewed changes

capri-xiyue added 2 commits November 17, 2025 10:37

rename and refactor

200dbf4

Signed-off-by: Xiyue Yu <[email protected]>

added epp name in env

9d1514a

capri-xiyue added 2 commits November 18, 2025 13:47

fixed import

72bb590

updated to use epp name instead of pod name

71f9fe5

kfswain reviewed Nov 18, 2025

View reviewed changes

capri-xiyue added 4 commits November 18, 2025 14:09

fixed compiler

770f96c

verify

a96edb0

don't set endpointpool in datastore for inferencepool at start

1f15673

rename endpoints to endpointsmeta

d079423

capri-xiyue requested a review from kfswain November 18, 2025 23:43

capri-xiyue added 3 commits November 18, 2025 15:45

rename import package

76b4eaa

rename test utility

2c67bf8

added logging info

7798ebd

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 19, 2025

resolve merge conflicts

1167364

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 19, 2025

ahg-g reviewed Nov 19, 2025

View reviewed changes

capri-xiyue added 2 commits November 20, 2025 11:16

change endpointpool struct

e33233b

fixed variable naming

86ac1f8

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 20, 2025

fixed linter

11a8a68

capri-xiyue requested a review from ahg-g November 20, 2025 22:28

k8s-ci-robot assigned ahg-g Nov 20, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 20, 2025

k8s-ci-robot merged commit 3c8aba1 into kubernetes-sigs:main Nov 20, 2025
11 checks passed

capri-xiyue mentioned this pull request Nov 20, 2025

Standalone EPP - Endpoint Discovery Using Pod Selector Investigation #1779

Open

Enable EPP to support endpoint discovery using pod selector #1833

Enable EPP to support endpoint discovery using pod selector #1833

Uh oh!

Conversation

capri-xiyue commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

capri-xiyue commented Nov 7, 2025

Uh oh!

elevran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

capri-xiyue commented Nov 14, 2025

Uh oh!

capri-xiyue commented Nov 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elevran commented Nov 19, 2025

Uh oh!

ahg-g commented Nov 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahg-g commented Nov 20, 2025

Uh oh!

k8s-ci-robot commented Nov 20, 2025

Uh oh!

ahg-g commented Nov 20, 2025

Uh oh!

ahg-g commented Nov 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

capri-xiyue commented Nov 7, 2025 •

edited

Loading

netlify bot commented Nov 7, 2025 •

edited

Loading