Skip to content

Conversation

@huydhn
Copy link
Contributor

@huydhn huydhn commented Oct 8, 2024

Do the same for iOS benchmark, there is no JSON output atm, so I parse the test spec output directly

Testing

https://github.com/pytorch/executorch/actions/runs/11239684907

@pytorch-bot
Copy link

pytorch-bot bot commented Oct 8, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5982

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c569208 with merge base 57e3c81 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 8, 2024
@huydhn huydhn temporarily deployed to upload-benchmark-results October 8, 2024 17:09 — with GitHub Actions Inactive
@huydhn huydhn requested review from guangy10 and shoumikhin October 8, 2024 20:07
@huydhn huydhn marked this pull request as ready for review October 8, 2024 20:07
@facebook-github-bot
Copy link
Contributor

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

--output-dir benchmark-results \
--repo ${{ github.repository }} \
--head-branch ${{ github.head_ref || github.ref_name }} \
--workflow-name ${{ github.workflow }} \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
--workflow-name ${{ github.workflow }} \
--workflow-name "${{ github.workflow }}" \

The workflow name must be passed as a string, otherwise you may see invalid arguments error

@guangy10
Copy link
Contributor

guangy10 commented Oct 8, 2024

there is no JSON output atm, so I parse the test spec output directly

Do you plan to add the JSON output?

@huydhn
Copy link
Contributor Author

huydhn commented Oct 8, 2024

Do you plan to add the JSON output?

I will need to sync with @shoumikhin on if it's possible to achieve that with the current form the the benchmark

@guangy10
Copy link
Contributor

guangy10 commented Oct 8, 2024

fyi, #6023 is putting up some fixes to include "backend" and "dtype" in the exported model name. Can't test it e2e for iOS since this PR is still WIP.

@guangy10
Copy link
Contributor

guangy10 commented Oct 8, 2024

@huydhn Can you schedule a run to include a non-genai model w/ coreml delegate, e.g. mv3? The coreml model will be named as mv3_coreml_all_fp16.pte, where "coreml_all" refers to the type name of the delegate and compute unit (e.g. ALL, CPU_ONLY, CPU_AND_GPU, CPU_AND_NE). Our models are taking ALL by default.

@huydhn
Copy link
Contributor Author

huydhn commented Oct 9, 2024

Another test with stories110M and mv2 https://github.com/pytorch/executorch/actions/runs/11245821876 with coreml

@huydhn huydhn had a problem deploying to upload-benchmark-results October 9, 2024 00:18 — with GitHub Actions Failure
@facebook-github-bot
Copy link
Contributor

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@huydhn merged this pull request in 192ca82.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants