Upload Android benchmark results to OSS benchmark database #5808
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5808
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit c942ab4 with merge base 2f9f94a ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
3 tasks
huydhn
added a commit
to pytorch/test-infra
that referenced
this pull request
Oct 3, 2024
This updates the mobile job to return the list of artifacts from AWS to the caller to use as they see fit. One of the use case is for ExecuTorch to extract the benchmark results JSON from `CUSTOMER_ARTIFACT` and `TESTSPEC_OUTPUT` (2 types of artifacts from AWS). My plan here is to: * [x] Get the list of artifacts (this PR) * [ ] On ExecuTorch side, extract the benchmark JSON from the artifacts pytorch/executorch#5808 * [ ] Create a new GitHub action on test-infra to upload the JSON (This could be done on ExecuTorch side too, but I plan to make this generic and reusable by other projects, so it must be on test-infra) Minor fixes: * Also update the console output to print out the context of `TESTSPEC_OUTPUT`, which is basically the main output from the device. * Print additional information about the job name (the device name from AWS in disguise) and app type (Android, iOS) * Some lint fix here and there.
4 tasks
Contributor
|
@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
kirklandsign
approved these changes
Oct 3, 2024
guangy10
reviewed
Oct 4, 2024
guangy10
reviewed
Oct 4, 2024
guangy10
reviewed
Oct 4, 2024
huydhn
had a problem deploying
to
upload-benchmark-results
GitHub Actions
Failure
huydhn
had a problem deploying
to
upload-benchmark-results
GitHub Actions
Failure
huydhn
had a problem deploying
to
upload-benchmark-results
GitHub Actions
Failure
huydhn
temporarily deployed
to
upload-benchmark-results
GitHub Actions
Inactive
Contributor
Author
|
The testing looks good https://github.com/pytorch/executorch/actions/runs/11189433899/job/31111425608, I can see the data on the database. |
huydhn
added a commit
to pytorch/test-infra
that referenced
this pull request
Oct 5, 2024
This adds a GHA to upload benchmark results. I'm using the existing `torchci-oss-ci-benchmark` database for now until the new generic database from https://fburl.com/gdoc/ossgtvte is ready. So, it's backed by dynamoDB instead of S3 for now. I also include a quick test with some dummy Android benchmark records. * [x] Get the list of artifacts #5727 * [x] On ExecuTorch side, extract the benchmark JSON from the artifacts pytorch/executorch#5808 * [x] Create a new GitHub action on test-infra to upload the JSON #5742 * [x] Create a new GHA role to upload the results meta-pytorch/pytorch-gha-infra#483
Contributor
|
@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
huydhn
temporarily deployed
to
upload-benchmark-results
GitHub Actions
Inactive
Summary: This PR adds a job to upload Android benchmark results to the benchmark database. It transforms the `benchmark_results.json` file slightly to fit into the current schema. We are going to have a better schema soon https://fburl.com/gdoc/ossgtvte, but landing this first would unblock the work on building the dashboard before the launch. Updating the schema can be done later. * The job processes what it finds, so if one model fails, the rest will still be uploaded. * I will follow up with another PR for iOS later. No need to wait for the TPS metric there, we'll upload what available first. There are still some TODO pending: * pytorch/test-infra#5742 * meta-pytorch/pytorch-gha-infra#483 But the structure of the CI job is ready to review. Pull Request resolved: #5808 Reviewed By: guangy10, kirklandsign Differential Revision: D63869876 Pulled By: huydhn
huydhn
force-pushed
the
upload-android-perf-results
branch
from
542db36 to
c942ab4
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D63869876 |
Contributor
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a job to upload Android benchmark results to the benchmark database. It transforms the
benchmark_results.jsonfile slightly to fit into the current schema. We are going to have a better schema soon https://fburl.com/gdoc/ossgtvte, but landing this first would unblock the work on building the dashboard before the launch. Updating the schema can be done later.There are still some TODO pending:
But the structure of the CI job is ready to review.