Handle model state in benchmark CI jobs

[metascroy]

Previously when running a pte file that has model state, our CI would fail with an error like:

2024-12-19 14:15:43.359148-0800 Benchmark[532:16686] [coreml] The input feature for layers_10_attention_SDPA_kv_cache_v_cache must be an MLState, but it was not.

This is because if you build the xcode frameworks with xcode 15, the ExecuTorch CoreML backend is built to not handle MLState variables:

executorch/backends/apple/coreml/runtime/delegate/MLModel_Prewarm.h

Line 11 in fc42a4e

#if !defined(MODEL_STATE_IS_SUPPORTED) && __has_include(<CoreML/MLModel+MLState.h>)

Note the prebuilt frameworks cannot handle MLState (https://github.com/pytorch/executorch/tree/main/extension/benchmark/apple/Benchmark#set-up-the-frameworks).

This PR:

• Edits the benchmark README.md to mention that Xcode 16.0 must be used to build frameworks that handle pte files that use CoreML MLState
• Reenables using MLState for iOS18 models

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7432

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures

As of commit e195698 with merge base a727b55 ():

NEW FAILURES - The following jobs have failed:

• apple-perf / benchmark-on-device (llama, coreml_fp16, apple_iphone_15, arn:aws:devicefarm:us-west-2:3085353851... / mobile-job (ios) (gh)
  Process completed with exit code 22.
• apple-perf / benchmark-on-device (llama, mps, apple_iphone_15, arn:aws:devicefarm:us-west-2:308535385114:devic... / mobile-job (ios) (gh)
  Process completed with exit code 22.
• apple-perf / benchmark-on-device (llama, xnnpack_q8, apple_iphone_15, arn:aws:devicefarm:us-west-2:30853538511... / mobile-job (ios) (gh)
  Process completed with exit code 22.
• apple-perf / build-benchmark-app / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 70
• apple-perf / upload-benchmark-results (gh)
  Process completed with exit code 1.
• Check Labels / Check labels (gh)
  RuntimeError: Error checking labels: PR does not have required labels

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[metascroy]

@guangy10 the apple-perf/build-benchmark-app job fails with "xcodebuild: error: Unable to find a device matching the provided destination specifier:2025-01-02T18:01:02.3778750Z { platform:iOS }"

Do you know what could cause this?

[guangy10]

Thanks @metascroy for improvement!

[guangy10]

This runner is configured with Xcode 16.0 right? Does it work for all other models that running on iOS 17?

[metascroy]

Most runners should have xcode 16, but you have to set the version line I do on line 333.

[metascroy]

Thanks @metascroy for improvement!

@guangy10 do you understanding the failing build-benchmark-app errror in CI: apple-perf/build-benchmark-app job fails with "xcodebuild: error: Unable to find a device matching the provided destination specifier:2025-01-02T18:01:02.3778750Z { platform:iOS }"

[guangy10]

Thanks @metascroy for improvement!

@guangy10 do you understanding the failing build-benchmark-app errror in CI: apple-perf/build-benchmark-app job fails with "xcodebuild: error: Unable to find a device matching the provided destination specifier:2025-01-02T18:01:02.3778750Z { platform:iOS }"

Can you share the job link? Can't find this error in the failed job. However, I'm seeing the missing stdlib errors again in pretty much all app builds, for example, https://github.com/pytorch/executorch/actions/runs/12586454085/job/35080314575?pr=7432 Maybe @shoumikhin can help with it?

[guangy10]

@guangy10 the apple-perf/build-benchmark-app job fails with "xcodebuild: error: Unable to find a device matching the provided destination specifier:2025-01-02T18:01:02.3778750Z { platform:iOS }"

Do you know what could cause this?

The required SDKROOT = iphoneos18.2, but no iPhone 15 from the device pool are equipped with 18.2? I'm not sure how the "Available destinations for the Benchmark scheme" is generated. @huydhn @shoumikhin do you know?

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.