Skip to content

Conversation

nirrozenbaum
Copy link
Contributor

@nirrozenbaum nirrozenbaum commented Aug 4, 2025

This PR aims to improve the request handling flow.
Before this PR the flow is:

  • director gets an incoming request. could be with subset filtering (only subset of the endpoints are candidates to serve the request)
  • saturation detector runs for all pods always, even if subset filtering was specified. as part of this code, saturation detection gets all pods from datastore.
  • director goes again to datastore to get candidate pods for scheduling while considering the subset filter header.
  • schedule
  • ....
    in this flow pods are read from the datastore twice in a row, and saturation check is done for all the pods even if subset filter was specified.

After this PR:

  • director gets an incoming request. could be with subset filtering (only subset of the endpoints are candidates to serve the request).
  • director goes to datastore to get candidate pods for scheduling while considering the subset filter header.
  • saturation detector runs only for the candidate pods for scheduling. it gets as input the candidate pods. no need to have datastore as a dependency in saturation detector.
  • if candidate pods are not saturated, continue to schedule
  • ...
    in this flow pods are read from the datastore once for the request flow and saturation check is done only for the relevant pods for the request.

tests updated accordingly.

Additionally, this PR removes the extensive documentation in some places. IMO we don't want to document every function internal step. this is not maintainable. the code should be self explainable (which it is!) in a way that doesn't require this extensive documentation. for example, in HandleRequest function (inside director), there is no need to explain in the godoc what are the steps. one can just read the function and easily understand the steps.

cc: @kfswain @LukeAVanDrie

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nirrozenbaum

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested review from ahg-g and liu-cong August 4, 2025 12:22
@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 4, 2025
Copy link

netlify bot commented Aug 4, 2025

Deploy Preview for gateway-api-inference-extension ready!

Name Link
🔨 Latest commit 76412f3
🔍 Latest deploy log https://app.netlify.com/projects/gateway-api-inference-extension/deploys/68a4698aeb77020008f90d6d
😎 Deploy Preview https://deploy-preview-1293--gateway-api-inference-extension.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

result, err := d.scheduler.Schedule(ctx, reqCtx.SchedulingRequest, candidatePods)

// Admission Control check
if err := d.admitRequest(ctx, candidatePods, requestCriticality, reqCtx.FairnessID); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could move admiRequest logic into getCandidatePodsForScheduling. Intuitively, the admiRequest function should be called before the getCandidatePodsForScheduling function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the main logic currently in admitReqeust is to check saturation of the system.
it's not relevant to check saturation of all the pods but only of the candidate pods (If we use subset filtering to specify list of pods and all of them are saturated, it's not helpful that other pods are available).

therefore, getCandidatePods is done before the admitRequest function.

think about it this way -
getCandidatePods func returns the set of pods that are relevant for current request.
admitRequest func check if any of the pods from the above set can actually serve the request, or if all of them are saturated.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 9, 2025
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 17, 2025
Copy link
Collaborator

@kfswain kfswain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some style based comments.

Once Flow Control is implemented, if we don't substantially expand the saturation detection logic, I think we should fold it into another package, I dont think there is enough logic to warrant its own package. But that's out of scope for this PR

result, err := d.scheduler.Schedule(ctx, reqCtx.SchedulingRequest, candidatePods)

// Admission Control check
if err := d.admitRequest(ctx, candidatePods, *infObjective.Spec.Priority, reqCtx.FairnessID); err != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that this dedupes the candidatePods == 0 check that was in the admitRequest func

Signed-off-by: Nir Rozenbaum <[email protected]>
@nirrozenbaum
Copy link
Contributor Author

nirrozenbaum commented Aug 19, 2025

Left some style based comments.

addressed the comments.

Once Flow Control is implemented, if we don't substantially expand the saturation detection logic, I think we should fold it into another package, I dont think there is enough logic to warrant its own package. But that's out of scope for this PR

yeah, I agree. it would probably go eventually into flow control or requestcontrol package.

Another "not in scope" comment - I think the saturation check itself should be an extension point and current code may be shipped with IGW as the default implementation of that extension point.
whoever is using IGW may want to define different criteria for saturation and not necessarily these metrics.
spinned an issue #1405.

Comment on lines 163 to 188
// admitRequest handles admission control to decide whether or not to accept the request
// based on the request priority and system saturation state.
func (d *Director) admitRequest(ctx context.Context, candidatePods []backendmetrics.PodMetrics,
requestPriority int, fairnessID string) error {
loggerTrace := log.FromContext(ctx).V(logutil.TRACE)

loggerTrace.Info("Entering Flow Control", "priority", requestPriority, "fairnessID", fairnessID)

// This will be removed in favor of a more robust implementation (Flow Control) in the very near future.
// TODO: Make this a configurable value.
// Tracking issue https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/1347
if requestPriority >= 0 {
loggerTrace.Info("Non-sheddable request bypassing saturation check.")
return nil
}

if d.saturationDetector.IsSaturated(ctx, candidatePods) {
return errutil.Error{
Code: errutil.InferencePoolResourceExhausted,
Msg: "system saturated, sheddable request dropped",
}
}

return nil
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this change I only switched the order of functions to be the same as the order they are called.

@kfswain
Copy link
Collaborator

kfswain commented Aug 26, 2025

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 26, 2025
@k8s-ci-robot k8s-ci-robot merged commit 7d84fb9 into kubernetes-sigs:main Aug 26, 2025
10 checks passed
@nirrozenbaum nirrozenbaum deleted the pod-list-once branch August 26, 2025 17:09
kfswain pushed a commit to kfswain/llm-instance-gateway that referenced this pull request Sep 1, 2025
…s#1293)

* remove datastore dependency from saturation detector

Signed-off-by: Nir Rozenbaum <[email protected]>

* addressed code review comments

Signed-off-by: Nir Rozenbaum <[email protected]>

---------

Signed-off-by: Nir Rozenbaum <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants