Skip to content

Commit 4612e09

Browse files
tmoreau89tmoreau89
committed
Llama3 on Android guide
1 parent 9478e20 commit 4612e09

File tree

2 files changed

+170
-0
lines changed

2 files changed

+170
-0
lines changed

‎recipes/inference/mobile_inference/android_inference/README.md

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# Running Llama3 8B Instruct on Android with MLC-LLM
2+
3+
# Overview
4+
In this tutorial we'll learn how to deploy Llama3 8B Instruct on an Android-based phone using MLC-LLM.
5+
6+
Machine Learning Compilation for Large Language Models (MLC LLM) is a high-performance universal deployment solution that allows native deployment of any large language models with native APIs with compiler acceleration. The mission of this project is to enable everyone to develop, optimize and deploy AI models natively on everyone's devices with ML compilation techniques.
7+
8+
You can read more about MLC-LLM at the following [link](https://github.com/mlc-ai/mlc-llm).
9+
10+
This tutorial was tested with the following setup:
11+
* MacBook Pro 16 inch from 2021 with Apple M1 Max and 32GB of RAM running Sonoma 14.3.1
12+
* OnePlus 12 Android Smartphone with a Snapdragon 8Gen3 SoC and 12GB or RAM, running OxygenOS 14.0
13+
14+
Running Llama3 on a phone will likely require a powerful chipset. We haven't tested extensively the range of chipset that will support this usecase. Feel free to update this README.md to specify what devices were successfully tested.
15+
16+
| Phone | Chipset | RAM | Status | Comments |
17+
|------------|------------------|------|---------|----------|
18+
| OnePlus 12 | Snapdragon 8Gen3 | 12GB | Success | None |
19+
| | | | | |
20+
21+
This guide is heavily based on the [MLC Android Guide](https://llm.mlc.ai/docs/deploy/android.html), but several steps have been taken to streamline the instructions.
22+
23+
# Pre-requisites
24+
25+
## Python
26+
27+
Whether you're using conda or virtual env to manage your environment, we highly recommend starting from scratch with a clean new environment.
28+
29+
For instance with virtual environment:
30+
```bash
31+
python3 -m venv .venv
32+
source .venv/bin/activate
33+
```
34+
35+
Next you'll need to install the following packages:
36+
```bash
37+
python3 -m pip install -r requirements.txt
38+
```
39+
40+
## Rust
41+
42+
[Rust](https://www.rust-lang.org/tools/install) is needed to cross-compile HuggingFace tokenizers to Android.
43+
Make sure rustc, cargo, and rustup are available in $PATH.
44+
45+
46+
## Android Studio
47+
48+
Android Studio [install](https://developer.android.com/studio) with NDK and CMake.
49+
50+
To install NDK and CMake, in the Android Studio welcome page, click “Projects → SDK Manager → SDK Tools”. Set up the following environment variables:
51+
52+
* ANDROID_NDK so that $ANDROID_NDK/build/cmake/android.toolchain.cmake is available.
53+
* TVM_NDK_CC that points to NDK’s clang compiler.
54+
55+
For instance, the paths will look like the following on OSX for user `moreau`:
56+
```bash
57+
# Android + TVM setup
58+
export ANDROID_NDK="/Users/moreau/Library/Android/sdk/ndk/26.1.10909125"
59+
export TVM_NDK_CC="$ANDROID_NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android24-clang"
60+
```
61+
62+
This tutorial was tested successfully on Android Studio Hedgehog | 2023.1.1 Patch 1.
63+
64+
## JDK
65+
66+
JDK, such as OpenJDK >= 17, to compile Java bindings of TVM Unity runtime.
67+
68+
We strongly recommend setting the JAVA_HOME to the JDK bundled with Android Studio. Using Android Studio’s JBR bundle as recommended here https://developer.android.com/build/jdks will reduce the chances of potential errors in JNI compilation.
69+
70+
For instance on macOS, you'll need to point JAVA_HOME to the following.
71+
72+
```bash
73+
export JAVA_HOME=/Applications/Android\ Studio.app/Contents/jbr/Contents/Home
74+
```
75+
76+
To make sure the java binary can be found do an `ls $JAVA_HOME/bin/java`
77+
78+
## MLC-LLM
79+
80+
Let's clone mlc-llm from its repo in the directory of your choice:
81+
82+
```bash
83+
cd /path/to/where/to/clone/repo
84+
git clone https://github.com/mlc-ai/mlc-llm --recursive
85+
export MLC_LLM_HOME=/path/to/mlc-llm
86+
```
87+
88+
At the time of writing this README, we tested `mlc-llm` at the following sha: `21feb7010db02e0c2149489f5972d6a8a796b5a0`.
89+
90+
## Phone Setup
91+
92+
On your phone, enable debugging on your phone in your phone’s developer settings. Each phone manufacturer will have its own approach to enabling debug mode, so a simple Google search should equip you with the steps to do that on your phone.
93+
94+
In addition, make sure to change your USB configuration from "Charging" to "MTP (Media Transfer Protocol)". This will allow us to connect to the device serially.
95+
96+
Connect your phone to your development machine. On OSX, you'll be prompted on the dev machine whether you want to allow the accessory to connect. Hit "Allow".
97+
98+
# Build Steps
99+
100+
First edit the file under `android/MLCChat/mlc-package-config.json` and replace the contents with the following:
101+
102+
```json
103+
{
104+
"device": "android",
105+
"model_list": [
106+
{
107+
"model": "HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC",
108+
"estimated_vram_bytes": 4348727787,
109+
"model_id": "Llama-3-8B-Instruct",
110+
"overrides": {
111+
"context_window_size": 768,
112+
"prefill_chunk_size": 256
113+
}
114+
}
115+
]
116+
}
117+
```
118+
119+
To understand what these JSON fields mean you can refer to this [documentation](https://llm.mlc.ai/docs/deploy/android.html#step-2-build-runtime-and-model-libraries).
120+
121+
122+
From the `mlc-llm` project root directory:
123+
124+
```bash
125+
cd $MLC_LLM_HOME
126+
cd android/MLCChat
127+
python3 -m mlc_llm package --package-config mlc-package-config.json --output dist
128+
```
129+
130+
The command above will take a few minutes to run as it runs through the following steps:
131+
132+
* Compile the Llama 3 8B instruct specified in the `mlc-package-config.json` into a binary model library.
133+
* Build the `mlc-llm` runtime and tokenizer. In addition to the model itself, a lightweight runtime and tokenizer are required to actually run the LLM.
134+
135+
Now let's launch Android Studio.
136+
137+
* On the "Welcome to Android Studio" page, hit "Open", and navigate to `$MLC_LLM_HOME/android/MLCChat`, then hit "Open"
138+
* A window will pop up asking whether to "Trust and Open project 'MLCChat'" - hit "Trust Project"
139+
* The project will now launch
140+
* Under File -> Project Structure... -> Project change the Gradle Version (second drop down from the top) to 8.5
141+
142+
Connect your phone to your development machine - assuming you've followed the setup steps in the pre-requisite section, you should be able to see the device.
143+
144+
Next:
145+
146+
* Hit Build -> Make Project.
147+
* Hit Run -> Run 'app'
148+
149+
The MLCChat app will launch on your phone, now access your phone:
150+
151+
* Under Model List you'll see the `Llama-3-8B-Instruct` LLM listed.
152+
* The model's not quite ready to launch yet, because the weights need to be downloaded over Wifi first. Hit the Download button on the right to the model name to download the weights from HuggingFace.
153+
154+
Note that you can change the build settings to bundle the weights with the MLCChat app so you don't have to download the weights over wifi. To do so you can follow the instructions [here](https://llm.mlc.ai/docs/deploy/android.html#bundle-model-weights).
155+
156+
Once the model weights are downloaded you can now interact with Llama 3 locally on your Android phone!

‎recipes/inference/mobile_inference/android_inference/requirements.txt

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
--pre
2+
--find-links https://mlc.ai/wheels
3+
mlc-llm-nightly
4+
mlc-ai-nightly
5+
attrs
6+
decorator
7+
numpy
8+
psutil
9+
pydantic
10+
requests
11+
scipy
12+
setuptools
13+
torch
14+
tqdm

0 commit comments

Comments
 (0)