You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: examples/models/llama2/README.md
+6Lines changed: 6 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -332,6 +332,8 @@ Note for Mac users: There's a known linking issue with Xcode 15.1. Refer to the
332
332
333
333
For Llama2 and stories models, pass the converted `tokenizer.bin` file instead of `tokenizer.model`.
334
334
335
+
To build for CoreML backend and validate on Mac, replace `-DEXECUTORCH_BUILD_XNNPACK=ON` with `-DEXECUTORCH_BUILD_COREML=ON`
336
+
335
337
## Step 5: Run benchmark on Android phone
336
338
337
339
**1. Build llama runner binary for Android**
@@ -418,6 +420,10 @@ for each backend ([CoreML](https://pytorch.org/executorch/main/build-run-coreml.
418
420
419
421
The iOS LLAMA app supports the CoreML and MPS model and the Android LLAMA app supports the QNN model. On Android, it also allow to cross compiler the llama runner binary, push to the device and run.
420
422
423
+
For CoreML, there are 2 additional optional arguments:
424
+
* `--coreml-ios`: Specify the minimum iOS version to deploy (and turn on available optimizations). E.g. `--coreml-ios 18` will turn on [in-place KV cache](https://developer.apple.com/documentation/coreml/mlstate?language=objc) and [fused scaled dot product attention kernel](https://apple.github.io/coremltools/source/coremltools.converters.mil.mil.ops.defs.html#coremltools.converters.mil.mil.ops.defs.iOS18.transformers.scaled_dot_product_attention) (the resulting model will then need at least iOS 18 to run, though)
425
+
* `--coreml-quantize`: Use [quantization tailored for CoreML](https://apple.github.io/coremltools/docs-guides/source/opt-quantization-overview.html). E.g. `--coreml-quantize b4w` will perform per-block 4-bit weight-only quantization in a way tailored for CoreML
426
+
421
427
# What is coming next?
422
428
## Quantization
423
429
- Enabling FP16 model to leverage smaller groupsize for 4-bit quantization.
0 commit comments