v1.0 InferencePool API Review #1173

capri-xiyue · 2025-07-16T19:06:41Z

What type of PR is this?
/kind api-change

What this PR does / why we need it:
This PR is a diff of /apis from alpha (main branch) to v1.0 (release-1.0 branch). The InferencePool SPEC doesn't have any change except the group change from inference.networking.x-k8s.io to `inference.networking.k8s.io

Note: This PR is purely to facilitate review, it is not intended to merge.

To do the api review, please select the specific commit as the screenshot below so that you can just review the api related change

/assign @robscott

…-sigs#1160) Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.15.0 to 0.16.0. - [Commits](golang/sync@v0.15.0...v0.16.0) --- updated-dependencies: - dependency-name: golang.org/x/sync dependency-version: 0.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

This commit introduces a new pluggable framework for request queuing within the EPP Flow Control layer. This change establishes the core interfaces and initial implementations needed for sophisticated request management, prioritization, and fairness. The key components of this framework are: - **`framework.SafeQueue` Interface**: A new contract for concurrent-safe queue implementations. It defines a standard set of behaviors for adding, removing, peeking, and managing items, ensuring that all queue plugins are interchangeable. - **Queue Plugin Implementations**: - **`listqueue`**: A simple, efficient FIFO queue based on `container/list`. Ideal for basic, fair queuing workloads. - **`maxminheap`**: A priority queue based on a max-min heap, allowing for O(1) access to both the highest and lowest priority items. This is suitable for advanced policies that require configurable ordering. - **Plugin Registration**: A factory pattern (`queue.MustRegisterQueue`) allows new queue implementations to be discovered and registered at runtime, making the system extensible. - **Comprehensive Testing**: - A new conformance test suite (`TestQueueConformance`) ensures that all registered queue plugins strictly adhere to the `SafeQueue` contract, covering lifecycle, ordering, edge cases, and concurrency. - A centralized benchmark suite (`BenchmarkQueues`) provides a fair, apples-to-apples performance comparison of all queue implementations across various workload patterns. - **Core Type Refinements**: The `types` package has been updated to support this new framework, including a refined `QueueItemAccessor` interface and a new `QueueItemHandle` for opaque, safe item manipulation. This framework decouples the core flow control logic from the specific queuing disciplines, enabling future work on advanced dispatch and displacement policies.

…gs#1157) Signed-off-by: Nir Rozenbaum <[email protected]>

* Conformance: Fixes the EPP ConfigMap namespace Signed-off-by: Daneyon Hansen <[email protected]> * Renames config file in rollout.md Signed-off-by: Daneyon Hansen <[email protected]> --------- Signed-off-by: Daneyon Hansen <[email protected]>

This commit introduces the `IntraFlowDispatchPolicy` framework, the second major component of the new pluggable flow control system. This framework decouples the logic for selecting a request from within a single flow's queue (temporal scheduling) from the underlying queue data structure. Key components include: - `framework.IntraFlowDispatchPolicy`: The core interface that defines the contract for selecting an item from a flow's queue. - `framework.FlowQueueAccessor`: A read-only interface that provides policies with safe access to queue state. - `RequiredQueueCapabilities`: A mechanism for policies to declare their queue requirements (e.g., FIFO, priority-ordered), which are validated by the registry. - A factory and registration system for discovering and instantiating policy plugins by name. - A comprehensive conformance test suite to validate the contract for all policy plugins. - A foundational `FCFS` (First-Come, First-Served) policy as the first reference implementation. This work builds directly on the `SafeQueue` framework, enabling the development of sophisticated, policy-driven request prioritization and scheduling.

k8s-ci-robot · 2025-07-16T19:06:50Z

Hi @capri-xiyue. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

capri-xiyue · 2025-07-16T19:07:01Z

/hold
Don't merge as it is just for api review.

netlify · 2025-07-16T19:07:12Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`6c02159`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/687811ca8fb1b50008233e1f
😎 Deploy Preview	https://deploy-preview-1173--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

capri-xiyue · 2025-07-16T20:53:02Z

/hold, this should not get merged

nirrozenbaum · 2025-07-17T17:29:42Z

why is it pushed to release 0.5 branch?

capri-xiyue · 2025-07-17T17:33:40Z

why is it pushed to release 0.5 branch?

This is not meant to merge, just for api review. As #1116 get merged, If I point it to main, there won't be any difference.

danehans · 2025-07-17T18:21:22Z

v1api/v1/inferencepool_types.go

+	// EndpointPickerConfig specifies the configuration needed by the proxy to discover and connect to the endpoint
+	// picker service that picks endpoints for the requests routed to this pool.
+	EndpointPickerConfig `json:",inline"`


As we discussed during today's community meeting, we need to decide on refining the EndpointPickerConfig type. We originally chose this API structure to support surfacing config for future extensions, and inlining to simplify the UI. From my understanding, we cannot change the inline after the API goes GA, so we should either simplify the EPP config API surface or remove inlining.

cc: @robscott @ahg-g @smarterclayton

If we are removing this struct, we may want to rename extensionRef to make it explicit that this is an epp extension; also, do we want to make extensionRef a list (and add a type enum, with the only value possible now EPP) to allow potential expansion to other pool attached extensions?

we cannot change the inline after the API goes GA

As long as the user-facing API surface remains the same (the CRD in this case), we can make any changes we want to the underlying Go types. Although we generally try to modify changes to underlying Go types, the only compatibility guarantees with k8s APIs are to the CRDs/API specs themselves.

I think the actual API surface here is ~fine as is, but I don't mind removing the extra struct layer as we can always add it back later if/when we need it.

Discussed in community meeting. We will try to clean it up before 1.0 release. But it's not a blocker. If we can't get it done before 1.0 release, we need to create a issue for it and include it in future milestone.

This is the PR for the clean up #1324

v1api/v1/inferencepool_types.go

aojea · 2025-07-18T08:57:26Z

v1api/v1/inferencepool_types.go

+	// Cross namesoace selector is not supported.
+	//
+	// +kubebuilder:validation:Required
+	Selector map[LabelKey]LabelValue `json:"selector"`


Related issue where there seems to be some use cases for using the full selector feature because the map fall short kubernetes/kubernetes#48528, just for reference since it is not going to be possible to evolve later, but I imagine that you have already discussed this

This is an intentional omission as many implementations are stuck creating a Service behind the scenes to get EndpointSlices, so we want the selector that's specified here to be able to map to Service as long as that's a common implementation detail (I'd hope this would be temporary). It could be worth trying to find a way to structure this field in a way that allows for a more fully featured selector in the future though. I think that would mean copying the upstream LabelSelector type and temporarily omitting matchExpressions.

Curious what maintainers think here:
cc @danehans @nirrozenbaum @kfswain

I have a draft PR on this: #1330 . In it I simply 1. turn the map[LabelKey]LabelValue to a LabelSelector 2. let the LabelSelector struct contains the map[LabelKey]LabelValue.

I'm assuming we only want to do it in v1 because this is also a breaking change.

danehans · 2025-07-30T22:00:29Z

xref #1267 on whether destinationEndpointHintKey and destinationEndpointHintMetadataNamespace should be exposed via InferencePool.

danehans · 2025-07-30T22:40:12Z

InferencePool FailureMode is no longer functional for Envoy v1.35.0-based implementations due to #1266.

k8s-ci-robot · 2025-07-31T17:19:44Z

Keywords which can automatically close issues and at(@) or hashtag(#) mentions are not allowed in commit messages.

The list of commits with invalid commit messages:

ddca1df chore(deps): bump golang.org/x/sync from 0.15.0 to 0.16.0 (chore(deps): bump golang.org/x/sync from 0.15.0 to 0.16.0 #1160)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot · 2025-07-31T17:19:49Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: capri-xiyue
Once this PR has been reviewed and has the lgtm label, please assign danehans for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

danehans · 2025-07-31T17:41:51Z

xref #735 (comment) regarding how to communicate EPP cert info to the Gateway implementation.

danehans · 2025-07-31T18:46:54Z

xref #1273 for backend protocol selection. cc: @robscott

dependabot bot and others added 5 commits July 15, 2025 08:20

removed USE_STREAMING env var from conformance + tests (kubernetes-si…

9e50fa2

…gs#1157) Signed-off-by: Nir Rozenbaum <[email protected]>

k8s-ci-robot assigned robscott Jul 16, 2025

k8s-ci-robot added kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 16, 2025

k8s-ci-robot requested review from kfswain and nirrozenbaum July 16, 2025 19:06

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 16, 2025

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jul 16, 2025

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 16, 2025

capri-xiyue closed this Jul 16, 2025

capri-xiyue force-pushed the capri-xiyue/capri-xiyue-v1-api-review branch from 5317094 to 4ffb5f6 Compare July 16, 2025 20:36

k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 16, 2025

capri-xiyue reopened this Jul 16, 2025

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 16, 2025

capri-xiyue closed this Jul 16, 2025

capri-xiyue force-pushed the capri-xiyue/capri-xiyue-v1-api-review branch from 77371fe to 4ffb5f6 Compare July 16, 2025 20:53

k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 16, 2025

added api review

6c02159

capri-xiyue changed the base branch from release-0.5 to main July 17, 2025 16:42

capri-xiyue changed the base branch from main to release-1.0 July 17, 2025 16:42

capri-xiyue changed the base branch from release-1.0 to release-0.5 July 17, 2025 16:43

danehans reviewed Jul 17, 2025

View reviewed changes

aojea reviewed Jul 18, 2025

View reviewed changes

v1api/v1/inferencepool_types.go Outdated Show resolved Hide resolved

aojea reviewed Jul 18, 2025

View reviewed changes

v1api/v1/inferencepool_types.go Outdated Show resolved Hide resolved

aojea reviewed Jul 18, 2025

View reviewed changes

fixed typo

dce1aea

danehans mentioned this pull request Jul 31, 2025

Coordinate Gateway Implementations for InferencePool GA #1272

Open

danehans mentioned this pull request Jul 31, 2025

Inference: Add v1 InferencePool Support kgateway-dev/kgateway#11837

Open

This was referenced Aug 7, 2025

cleanup: simplify endpointpickerconfig #1324

Open

Clean up of EndpointPickerConfig #1326

Open

[InferencePool Update] Allows for a more fully featured selector in InferencePool #1327

Open

zetxqx mentioned this pull request Aug 7, 2025

Upgrade the inferencePool selector to a struct from a map. #1330

Open

delavet mentioned this pull request Aug 8, 2025

Support for v1 InferencePool AliyunContainerService/ack-gateway-api#2

Open

Xunzhuo mentioned this pull request Aug 11, 2025

feat: support inferencepool v1 envoyproxy/ai-gateway#1033

Draft

v1.0 InferencePool API Review #1173

Are you sure you want to change the base?

v1.0 InferencePool API Review #1173

Conversation

capri-xiyue commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jul 16, 2025

Uh oh!

capri-xiyue commented Jul 16, 2025

Uh oh!

netlify bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

capri-xiyue commented Jul 16, 2025

Uh oh!

nirrozenbaum commented Jul 17, 2025

Uh oh!

capri-xiyue commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danehans commented Jul 30, 2025

Uh oh!

danehans commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 31, 2025

Uh oh!

k8s-ci-robot commented Jul 31, 2025

Uh oh!

danehans commented Jul 31, 2025

Uh oh!

danehans commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

capri-xiyue commented Jul 16, 2025 •

edited

Loading

netlify bot commented Jul 16, 2025 •

edited

Loading

capri-xiyue commented Jul 17, 2025 •

edited

Loading

danehans commented Jul 31, 2025 •

edited

Loading