-
Notifications
You must be signed in to change notification settings - Fork 182
removed datastore dependency from saturation detector #1293
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
removed datastore dependency from saturation detector #1293
Conversation
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: nirrozenbaum The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
pkg/epp/requestcontrol/director.go
Outdated
result, err := d.scheduler.Schedule(ctx, reqCtx.SchedulingRequest, candidatePods) | ||
|
||
// Admission Control check | ||
if err := d.admitRequest(ctx, candidatePods, requestCriticality, reqCtx.FairnessID); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we could move admiRequest
logic into getCandidatePodsForScheduling
. Intuitively, the admiRequest
function should be called before the getCandidatePodsForScheduling
function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the main logic currently in admitReqeust
is to check saturation of the system.
it's not relevant to check saturation of all the pods but only of the candidate pods (If we use subset filtering to specify list of pods and all of them are saturated, it's not helpful that other pods are available).
therefore, getCandidatePods
is done before the admitRequest
function.
think about it this way -
getCandidatePods
func returns the set of pods that are relevant for current request.
admitRequest
func check if any of the pods from the above set can actually serve the request, or if all of them are saturated.
301159f
to
728fe9a
Compare
Signed-off-by: Nir Rozenbaum <[email protected]>
728fe9a
to
3615110
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left some style based comments.
Once Flow Control is implemented, if we don't substantially expand the saturation detection logic, I think we should fold it into another package, I dont think there is enough logic to warrant its own package. But that's out of scope for this PR
result, err := d.scheduler.Schedule(ctx, reqCtx.SchedulingRequest, candidatePods) | ||
|
||
// Admission Control check | ||
if err := d.admitRequest(ctx, candidatePods, *infObjective.Spec.Priority, reqCtx.FairnessID); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like that this dedupes the candidatePods == 0
check that was in the admitRequest func
Signed-off-by: Nir Rozenbaum <[email protected]>
addressed the comments.
yeah, I agree. it would probably go eventually into flow control or requestcontrol package. Another "not in scope" comment - I think the saturation check itself should be an extension point and current code may be shipped with IGW as the default implementation of that extension point. |
// admitRequest handles admission control to decide whether or not to accept the request | ||
// based on the request priority and system saturation state. | ||
func (d *Director) admitRequest(ctx context.Context, candidatePods []backendmetrics.PodMetrics, | ||
requestPriority int, fairnessID string) error { | ||
loggerTrace := log.FromContext(ctx).V(logutil.TRACE) | ||
|
||
loggerTrace.Info("Entering Flow Control", "priority", requestPriority, "fairnessID", fairnessID) | ||
|
||
// This will be removed in favor of a more robust implementation (Flow Control) in the very near future. | ||
// TODO: Make this a configurable value. | ||
// Tracking issue https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/1347 | ||
if requestPriority >= 0 { | ||
loggerTrace.Info("Non-sheddable request bypassing saturation check.") | ||
return nil | ||
} | ||
|
||
if d.saturationDetector.IsSaturated(ctx, candidatePods) { | ||
return errutil.Error{ | ||
Code: errutil.InferencePoolResourceExhausted, | ||
Msg: "system saturated, sheddable request dropped", | ||
} | ||
} | ||
|
||
return nil | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this change I only switched the order of functions to be the same as the order they are called.
/lgtm |
…s#1293) * remove datastore dependency from saturation detector Signed-off-by: Nir Rozenbaum <[email protected]> * addressed code review comments Signed-off-by: Nir Rozenbaum <[email protected]> --------- Signed-off-by: Nir Rozenbaum <[email protected]>
This PR aims to improve the request handling flow.
Before this PR the flow is:
in this flow pods are read from the datastore twice in a row, and saturation check is done for all the pods even if subset filter was specified.
After this PR:
in this flow pods are read from the datastore once for the request flow and saturation check is done only for the relevant pods for the request.
tests updated accordingly.
Additionally, this PR removes the extensive documentation in some places. IMO we don't want to document every function internal step. this is not maintainable. the code should be self explainable (which it is!) in a way that doesn't require this extensive documentation. for example, in HandleRequest function (inside director), there is no need to explain in the godoc what are the steps. one can just read the function and easily understand the steps.
cc: @kfswain @LukeAVanDrie