Skip to content

Commit 418c4c3

Browse files
yifan_shen3facebook-github-bot
authored andcommitted
Polish CoreML Llama Doc (pytorch#5745)
Summary: Document * The export python script arguments to export llama for coreml * The cmake argument to compile Mac runner with coreml backend Minor fix: Remove outdated arguments Pull Request resolved: pytorch#5745 Reviewed By: kirklandsign Differential Revision: D63652220 Pulled By: cccclai fbshipit-source-id: 1703158b4e2b31cc29d08e9262e56d564a90e539
1 parent 9c38cf7 commit 418c4c3

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

examples/models/llama2/README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -332,6 +332,8 @@ Note for Mac users: There's a known linking issue with Xcode 15.1. Refer to the
332332
333333
For Llama2 and stories models, pass the converted `tokenizer.bin` file instead of `tokenizer.model`.
334334
335+
To build for CoreML backend and validate on Mac, replace `-DEXECUTORCH_BUILD_XNNPACK=ON` with `-DEXECUTORCH_BUILD_COREML=ON`
336+
335337
## Step 5: Run benchmark on Android phone
336338
337339
**1. Build llama runner binary for Android**
@@ -418,6 +420,10 @@ for each backend ([CoreML](https://pytorch.org/executorch/main/build-run-coreml.
418420
419421
The iOS LLAMA app supports the CoreML and MPS model and the Android LLAMA app supports the QNN model. On Android, it also allow to cross compiler the llama runner binary, push to the device and run.
420422
423+
For CoreML, there are 2 additional optional arguments:
424+
* `--coreml-ios`: Specify the minimum iOS version to deploy (and turn on available optimizations). E.g. `--coreml-ios 18` will turn on [in-place KV cache](https://developer.apple.com/documentation/coreml/mlstate?language=objc) and [fused scaled dot product attention kernel](https://apple.github.io/coremltools/source/coremltools.converters.mil.mil.ops.defs.html#coremltools.converters.mil.mil.ops.defs.iOS18.transformers.scaled_dot_product_attention) (the resulting model will then need at least iOS 18 to run, though)
425+
* `--coreml-quantize`: Use [quantization tailored for CoreML](https://apple.github.io/coremltools/docs-guides/source/opt-quantization-overview.html). E.g. `--coreml-quantize b4w` will perform per-block 4-bit weight-only quantization in a way tailored for CoreML
426+
421427
# What is coming next?
422428
## Quantization
423429
- Enabling FP16 model to leverage smaller groupsize for 4-bit quantization.

0 commit comments

Comments
 (0)