removed datastore dependency from saturation detector #1293
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: nirrozenbaum The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
approved
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
pkg/epp/requestcontrol/director.go
Outdated
| result, err := d.scheduler.Schedule(ctx, reqCtx.SchedulingRequest, candidatePods) | ||
|
|
||
| // Admission Control check | ||
| if err := d.admitRequest(ctx, candidatePods, requestCriticality, reqCtx.FairnessID); err != nil { |
There was a problem hiding this comment.
Maybe we could move admiRequest logic into getCandidatePodsForScheduling. Intuitively, the admiRequest function should be called before the getCandidatePodsForScheduling function.
There was a problem hiding this comment.
the main logic currently in admitReqeust is to check saturation of the system.
it's not relevant to check saturation of all the pods but only of the candidate pods (If we use subset filtering to specify list of pods and all of them are saturated, it's not helpful that other pods are available).
therefore, getCandidatePods is done before the admitRequest function.
think about it this way -
getCandidatePods func returns the set of pods that are relevant for current request.
admitRequest func check if any of the pods from the above set can actually serve the request, or if all of them are saturated.
nirrozenbaum
force-pushed
the
pod-list-once
branch
from
301159f to
728fe9a
Compare
k8s-ci-robot
added
the
needs-rebase
Signed-off-by: Nir Rozenbaum <[email protected]>
nirrozenbaum
force-pushed
the
pod-list-once
branch
from
728fe9a to
3615110
Compare
k8s-ci-robot
removed
the
needs-rebase
There was a problem hiding this comment.
Left some style based comments.
Once Flow Control is implemented, if we don't substantially expand the saturation detection logic, I think we should fold it into another package, I dont think there is enough logic to warrant its own package. But that's out of scope for this PR
| result, err := d.scheduler.Schedule(ctx, reqCtx.SchedulingRequest, candidatePods) | ||
|
|
||
| // Admission Control check | ||
| if err := d.admitRequest(ctx, candidatePods, *infObjective.Spec.Priority, reqCtx.FairnessID); err != nil { |
There was a problem hiding this comment.
I like that this dedupes the candidatePods == 0 check that was in the admitRequest func
Signed-off-by: Nir Rozenbaum <[email protected]>
addressed the comments.
yeah, I agree. it would probably go eventually into flow control or requestcontrol package. Another "not in scope" comment - I think the saturation check itself should be an extension point and current code may be shipped with IGW as the default implementation of that extension point. |
| // admitRequest handles admission control to decide whether or not to accept the request | ||
| // based on the request priority and system saturation state. | ||
| func (d *Director) admitRequest(ctx context.Context, candidatePods []backendmetrics.PodMetrics, | ||
| requestPriority int, fairnessID string) error { | ||
| loggerTrace := log.FromContext(ctx).V(logutil.TRACE) | ||
|
|
||
| loggerTrace.Info("Entering Flow Control", "priority", requestPriority, "fairnessID", fairnessID) | ||
|
|
||
| // This will be removed in favor of a more robust implementation (Flow Control) in the very near future. | ||
| // TODO: Make this a configurable value. | ||
| // Tracking issue https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/1347 | ||
| if requestPriority >= 0 { | ||
| loggerTrace.Info("Non-sheddable request bypassing saturation check.") | ||
| return nil | ||
| } | ||
|
|
||
| if d.saturationDetector.IsSaturated(ctx, candidatePods) { | ||
| return errutil.Error{ | ||
| Code: errutil.InferencePoolResourceExhausted, | ||
| Msg: "system saturated, sheddable request dropped", | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
There was a problem hiding this comment.
In this change I only switched the order of functions to be the same as the order they are called.
|
/lgtm |
k8s-ci-robot
added
the
lgtm
…s#1293) * remove datastore dependency from saturation detector Signed-off-by: Nir Rozenbaum <[email protected]> * addressed code review comments Signed-off-by: Nir Rozenbaum <[email protected]> --------- Signed-off-by: Nir Rozenbaum <[email protected]>
4 participants
This PR aims to improve the request handling flow.
Before this PR the flow is:
in this flow pods are read from the datastore twice in a row, and saturation check is done for all the pods even if subset filter was specified.
After this PR:
in this flow pods are read from the datastore once for the request flow and saturation check is done only for the relevant pods for the request.
tests updated accordingly.
Additionally, this PR removes the extensive documentation in some places. IMO we don't want to document every function internal step. this is not maintainable. the code should be self explainable (which it is!) in a way that doesn't require this extensive documentation. for example, in HandleRequest function (inside director), there is no need to explain in the godoc what are the steps. one can just read the function and easily understand the steps.
cc: @kfswain @LukeAVanDrie