Skip to content

Handle model state in benchmark CI jobs #7432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Handle model state in benchmark CI jobs #7432

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
032ae0d
init
metascroy Dec 23, 2024
e5b7f2a
fix perf.yml
metascroy Dec 24, 2024
20f12ce
up
metascroy Dec 24, 2024
bcbdc0c
up
metascroy Dec 24, 2024
34861a5
xcode version
metascroy Dec 24, 2024
f38f4b5
doc up
metascroy Dec 24, 2024
46623d4
up
metascroy Dec 24, 2024
34c2ab7
up
metascroy Dec 24, 2024
1b876ca
up
metascroy Dec 24, 2024
a05b373
up
metascroy Dec 24, 2024
995ab67
Update apple-perf.yml
metascroy Jan 2, 2025
93be0ab
Update apple-perf.yml
metascroy Jan 2, 2025
e195698
Merge branch 'main' into xcode-16-for-ios-18
metascroy Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/apple-perf.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -341,7 +341,7 @@ jobs:
- set-parameters
secrets: inherit
with:
runner: macos-latest-xlarge
runner: macos-15-xlarge
Copy link
Contributor

@guangy10 guangy10 Jan 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This runner is configured with Xcode 16.0 right? Does it work for all other models that running on iOS 17?

Copy link
Contributor Author

@metascroy metascroy Jan 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most runners should have xcode 16, but you have to set the version line I do on line 333.

python-version: '3.11'
submodules: 'true'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
Expand All @@ -351,6 +351,9 @@ jobs:
script: |
set -eux

echo "Setting XCode version"
sudo xcode-select -switch /Applications/Xcode_16.2.app

echo "::group::Setting up CI environment"
.ci/scripts/setup-conda.sh

Expand Down
9 changes: 9 additions & 0 deletions extension/benchmark/apple/Benchmark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,15 @@ kernels_portable.xcframework
kernels_quantized.xcframework
```

#### MLState
The prebuilt frameworks do not work with CoreML models that use MLState, which is used in LLM models exported for iOS18. When executing the *.pte file, you will see an error like:

```
The input feature for X must be an MLState, but it was not.
```

If you encounter this issue, build the frameworks locally (option 2) using **XCode 16.0** or higher.

## Adding Models and Resources

Place your exported model files (`.pte`) and any other resources (e.g., `tokenizer.bin`) into the `extension/benchmark/apple/Benchmark/Resources` directory:
Expand Down
6 changes: 1 addition & 5 deletions extension/llm/export/partitioner_lib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,11 +158,7 @@ def _validate_ios_version() -> None:
op_linear_quantizer_config=op_linear_quantizer_config,
)

# ExecuTorch does not build CoreML delegate runtime to handle state
# when using OSS scripts, so we define take_over_mutable_buffer = False,
# even when target is iOS18
# take_over_mutable_buffer = minimum_deployment_target >= ct.target.iOS18
take_over_mutable_buffer = False
take_over_mutable_buffer = minimum_deployment_target >= ct.target.iOS18
return CoreMLPartitioner( # pyre-fixme[16]
compile_specs=compile_specs,
take_over_mutable_buffer=take_over_mutable_buffer,
Expand Down
Loading