Skip to content

Experimental SLO-Aware Routing and Latency Prediction #1568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 35 commits into from
Sep 15, 2025
Merged

Experimental SLO-Aware Routing and Latency Prediction #1568

merged 35 commits into from
Sep 15, 2025

Conversation

BenjaminBraunDev
Copy link
Contributor

Initial PR for SLO-Aware Routing. See #1323 for complete implementation details.

Working state for experimental testing and use, further refactoring planned before comprehensive unit and integration tests are written.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Sep 11, 2025
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 11, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @BenjaminBraunDev. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 11, 2025
@kfswain
Copy link
Collaborator

kfswain commented Sep 12, 2025

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 12, 2025
@kfswain
Copy link
Collaborator

kfswain commented Sep 15, 2025

pls fix the invalid commit messages
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenjaminBraunDev, kfswain

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 15, 2025
kaushikmitr and others added 9 commits September 15, 2025 20:55
@kaushikmitr @BenjaminBraunDev
seperate servers for training and prediction
ddbd7db
@kaushikmitr @BenjaminBraunDev
add latency predictor
86d9eb6 
put the predictor functions in director in a helper function

add scores to reqcxt

record prediction duration metrics

add prefix cache score to model input

slo based routing changes

retreive  request priority queue from the datastore

update scoring logic
@BenjaminBraunDev
better inital implemenation
6432af7 
Add scheduling profile, working state

remove latencypredictor from director

Move all latency prediction logic out of director and into scheduling profile. Make all Request/Response plugins take in RequestContext
@BenjaminBraunDev
progress towards fixing up merge conflicts from latency predictor merge
bcb83be
@BenjaminBraunDev
More refactor progress, fixing and adding tests
bdb1d57
@BenjaminBraunDev
working state, latency prediction
4f1f4ae
@BenjaminBraunDev
Clean up changes, remove unneeded files, working functionality withou…
2c099f6 
…t latency flag and scheduling plugins
@BenjaminBraunDev
Rebase cleanup, remove duplicate lines
2852eb5
@BenjaminBraunDev
Integrate new alpha-beta slo scoring into scoring plugin
fe82a14
BenjaminBraunDev and others added 19 commits September 15, 2025 20:55
@BenjaminBraunDev
Fix prefix cache scoring for slo-aware routing
d91834c
@BenjaminBraunDev
Add pycache or latency predictor to gitignore
47c86b0
@BenjaminBraunDev
Rebase with main
0cb3466
@BenjaminBraunDev
Fix prefix cache scoring being piped to latencyprediction_helper
6e2b6e1
@kaushikmitr @BenjaminBraunDev
add dependancies in scorer
8a8521c
@kaushikmitr @BenjaminBraunDev
chage to single profile
62d1479
@kaushikmitr @BenjaminBraunDev
chage to single profile
38ba84d
@kaushikmitr @BenjaminBraunDev
restore two profiles
e13b53b
@kaushikmitr @BenjaminBraunDev
restore two profiles
f65ed44
@kaushikmitr @BenjaminBraunDev
restore two profiles
40e3b79
@kaushikmitr @BenjaminBraunDev
update admit request to shed based on predictions
c712de9
@BenjaminBraunDev
add TODOs for future changes
772b6a0
@BenjaminBraunDev
Change artifact registry references to personal compiled images
b0e1f1d
@BenjaminBraunDev
Fix existing non-slo aware routing unit tests
ef504d9
@kaushikmitr @BenjaminBraunDev
update latency predictor with better eval metrics
82754ff
@BenjaminBraunDev
Fix saturation detector unit test
a7fd852
@BenjaminBraunDev
Change naming of SLO headers and prediction based routing header
efaced4
@BenjaminBraunDev
Remove port 9002 service on InferencePool causing make test to fail
ed28448
@BenjaminBraunDev
Fix epp hermetic integration test to expect ProcessingMode Send in re…
6ed7590 
…sponse header
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Sep 15, 2025
@BenjaminBraunDev
Copy link
Contributor Author

pls fix the invalid commit messages /approve

done

@kfswain
Copy link
Collaborator

kfswain commented Sep 15, 2025

/retest

This flake is fixed, you may want to consider pulling that into this PR but up to you all.

@BenjaminBraunDev
Copy link
Contributor Author

/retest

This flake is fixed, you may want to consider pulling that into this PR but up to you all.

Oh yeah, I know what this is. The weighted random test didn't use a consistent random seed and sometimes failed. I experienced this while testing locally.

@kfswain
Copy link
Collaborator

kfswain commented Sep 15, 2025

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 15, 2025
@k8s-ci-robot k8s-ci-robot merged commit db08577 into kubernetes-sigs:slo-prediction-experimental Sep 15, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nirrozenbaum nirrozenbaum Awaiting requested review from nirrozenbaum

@robscott robscott Awaiting requested review from robscott

Assignees

@kfswain kfswain

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@BenjaminBraunDev @k8s-ci-robot @kfswain @kaushikmitr