Skip to content

Conversation

@metascroy
Copy link
Contributor

This PR updates the alpha/beta CoreML model testing in a few ways:

  • All models are now testing with to_edge_lower_and_transform, the method we recommend in docs
  • We run pybindings on exported models to ensure 1) they run, and 2) the output is similar to pytorch eager
  • We add more model tests to CI

@metascroy metascroy requested a review from cccclai as a code owner July 24, 2025 05:09
@pytorch-bot
Copy link

pytorch-bot bot commented Jul 24, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12808

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 19 Pending

As of commit 4d7045f with merge base 2c84f70 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 24, 2025
@metascroy metascroy requested review from digantdesai and removed request for cccclai July 24, 2025 05:09
@github-actions
Copy link

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@metascroy metascroy requested a review from shoumikhin as a code owner July 24, 2025 18:12
@metascroy metascroy requested a review from cccclai July 24, 2025 21:02
@metascroy
Copy link
Contributor Author

@digantdesai @cccclai can I get a stamp?

# Build and test coreml model
MODELS=(mv3 ic4 resnet50 edsr mobilebert w2l)
for MODEL_NAME in "${MODELS[@]}"; do
for MODEL_NAME in dl3 edsr emformer_join ic3 ic4 mobilebert mv2 mv3 resnet50 vit w2l; do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@metascroy have you seen this PR?

https://github.com/pytorch/executorch/pull/12371/files

What are your thoughts on consolidating? I can drop my PR but maybe you can take a look and see if there are things worth incorporating?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mergennachin I missed that PR, but I can incorporate the following changes from your PR here:

  • Split CoreML/MPS into two separate jobs
  • Merge the model lists from the two PRs

For the runtime checks, I'll proceed with the pybind testing I'm adding here (vs. the C++ runner checking in your PR) because with pybinding we also check model accuracy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @metascroy

Changes look great

@metascroy metascroy force-pushed the update-coreml-model-testing branch from 1a9f07b to 4d7045f Compare July 28, 2025 17:47
@metascroy metascroy merged commit 37e3003 into main Jul 28, 2025
325 checks passed
@metascroy metascroy deleted the update-coreml-model-testing branch July 28, 2025 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants