NOCOMMIT: Bisect CoreML failures when bumping PyTorch pin

[swolchok]

trunk / test-llama-runner-mac (fp32, coreml) / is failing on #11154. trying to bisect to a PT commit per request from @metascroy

[swolchok]

Stack from ghstack (oldest at bottom):

• -> NOCOMMIT: Bisect CoreML failures when bumping PyTorch pin #11222

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11222

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures

As of commit 4061a9f with merge base 2e9c71c ():

NEW FAILURES - The following jobs have failed:

• Build Presets / apple (ios-simulator) / build (gh)
  The process '/opt/homebrew/bin/git' failed with exit code 128
• Build Presets / apple (ios) / build (gh)
• Check Labels / Check labels (gh)
  RuntimeError: Error checking labels: PR does not have required labels

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:.

If not, please add the release notes: none label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[swolchok]

bisects to 7e80f23516a86e18ae5bc5579d3005c1e7610102, which is pytorch/pytorch#153795
CC @angelayi

[metascroy]

bisects to 7e80f23516a86e18ae5bc5579d3005c1e7610102, which is pytorch/pytorch#153795 CC @angelayi

This is quite interesting. It's hard to see how @angelayi's PR could have affected the LLM model accuracy for CoreML. Are you sure on the bisect @swolchok?

[swolchok]

bisects to 7e80f23516a86e18ae5bc5579d3005c1e7610102, which is pytorch/pytorch#153795 CC @angelayi

This is quite interesting. It's hard to see how @angelayi's PR could have affected the LLM model accuracy for CoreML. Are you sure on the bisect @swolchok?

yes, unfortunately we don't have CI history for each version of this PR but I believe you can see the commit history. all I did was change the hash in this diff and record whether the job in the summary succeeded. I can repeat the test for the blame rev and previous rev to make sure it's not just flaky

[swolchok]

I confirm that test-llama-runner-mac (fp32, coreml) https://github.com/pytorch/executorch/actions/runs/15351527127/job/43200693055?pr=11222 fails with the blamed commit

[swolchok]

I confirm that test-llama-runner-mac (fp32, coreml) https://github.com/pytorch/executorch/actions/runs/15351973357/job/43202245242?pr=11222 succeeds with the commit immediately prior to the blamed commit. @metascroy bisect confirmed

[angelayi]

Is it possible to repro this accuracy issue locally? I'm a little confused why my changes would break the coreml test, unless coreml is somehow using aotinductor under the hood?

[swolchok]

Is it possible to repro this accuracy issue locally? I'm a little confused why my changes would break the coreml test, unless coreml is somehow using aotinductor under the hood?

@angelayi Looks like you run bash .ci/scripts/test_llama.sh -model stories110M -build_tool cmake -dtype fp32 -mode coreml with #11154 checked out and ExecuTorch installed. Verifying now

[swolchok]

I can't verify because backends/apple/coreml/scripts/install_requirements.sh is failing for me with build errors in numpy (e.g., std::ptrdiff_t not existing)

[swolchok]

I can't verify because backends/apple/coreml/scripts/install_requirements.sh is failing for me with build errors in numpy (e.g., std::ptrdiff_t not existing)

it is not necessary to run backends/apple/coreml/scripts/install_requirements.sh .

Failure did not repro locally.