testing: track the llama export times #15535
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15535
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 0855d71 with merge base 3dbc15b ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
Hi @mroreo! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
This PR needs a
|
meta-cla
bot
added
the
CLA Signed
mroreo
changed the title
| ;; | ||
| esac | ||
|
|
||
| MAX_EXPORT_TIME=$((ACT_EXPORT_TIME + BUFFER_TIME)) |
There was a problem hiding this comment.
nit: move this to line 149, after BUFFER_TIME, given that it is used on line 172 - causing errors on the latest run.
There was a problem hiding this comment.
Please tag/ping me again if approval is required to re-run the tests :/
There was a problem hiding this comment.
instead of moving it to line 149, since it depends on ACT_EXPORT_TIME, i just did the inline calculation in line 172
.ci/scripts/test_llama.sh
Outdated
|
|
||
| # Check export time against threshold. Default is 500 seconds. | ||
| if [ $EXPORT_DURATION -gt $MAX_EXPORT_TIME ]; then | ||
| echo "Failure; Export took ${EXPORT_DURATION} seconds, exceeding threshold of ${MAX_EXPORT_TIME} seconds" |
There was a problem hiding this comment.
Let's add some action item message too, like "This PR may have regressed export time; please review again or bump this threshold."
.ci/scripts/test_llama.sh
Outdated
| # macOS configurations | ||
| "macos:fp32:mps") ACT_EXPORT_TIME=30 ;; | ||
| "macos:fp32:coreml") ACT_EXPORT_TIME=61 ;; | ||
| "macos:fp32:xnnpack+custom+quantize_kv") ACT_EXPORT_TIME=133 ;; |
There was a problem hiding this comment.
Thank you for getting these baseline numbers!
|
@lucylq can you please help re-run the tests |
|
@lucylq can you help re-run this. I added more configurations and updated the times |
2 participants
Summary:
Based on the initial test times the following times were logged for each of the model export times from the following run
Run 1
pull - https://github.com/pytorch/executorch/actions/runs/19052918428
trunk - https://github.com/pytorch/executorch/actions/runs/19052918424
Run 2
pull - https://github.com/pytorch/executorch/actions/runs/19151256463
trunk - https://github.com/pytorch/executorch/actions/runs/19151256445
Fixes #10761
Test Plan:
Since I have no access to the github runners and the workflows, I dont have a test plan. Ideally, the github actions would be retriggered multiple times to track the maximum of the times across each trigger and then add an an arbitrary buffer value