Skip to content

Run benchmark test on PRs#229

Merged
jgchn merged 34 commits intollm-d:mainfrom
jgchn:pr-benchmark
Aug 8, 2025
Merged

Run benchmark test on PRs#229
jgchn merged 34 commits intollm-d:mainfrom
jgchn:pr-benchmark

Conversation

@jgchn
Copy link
Collaborator

@jgchn jgchn commented Aug 6, 2025

Fixes #227

  • Creates a kind cluster on GH Actions
  • Runs install_deps, standup, run (dry run since there is no harness endpoint), and teardown for a facebook/opt-125m model
  • Does not create PV and PVCs, does not require hf token, does not use cloud storage

jgchn added 30 commits August 6, 2025 17:47
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
jgchn added 4 commits August 8, 2025 11:05
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
Signed-off-by: Jing Chen <jing.chen2@ibm.com>
@jgchn jgchn marked this pull request as ready for review August 8, 2025 15:57
@jgchn jgchn requested a review from maugustosilva August 8, 2025 15:57
@maugustosilva
Copy link
Collaborator

This is pretty good, exactly what we needed. There are some improvements that should be added, such as including pvc creation test, but it can be done later.

@maugustosilva
Copy link
Collaborator

Question: how long does it take to create the kind cluster. The whole idea of this particular workflow is to be "as fast as possible", and we can use the pre-deployed cluster already present in benchmark1.yaml to save time.

@jgchn
Copy link
Collaborator Author

jgchn commented Aug 8, 2025

The whole workflow takes < 5min (check here). If we can use a pre-deployed cluster, that'd be great. But how would independent users have access to that cluster? I don't think PRs have permission to access secrets defined this workflow. But it's something to explore.

@maugustosilva
Copy link
Collaborator

You're correct, accessing the secrets from forks is not possible. LGTM and improve later.

@jgchn jgchn merged commit b025013 into llm-d:main Aug 8, 2025
2 checks passed
@jgchn jgchn deleted the pr-benchmark branch August 8, 2025 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve CI by running e2e workflow on PRs

2 participants