Skip to content

Add MediaTek Llama Runner in Android App Readme #6548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 29, 2024
Merged

Add MediaTek Llama Runner in Android App Readme #6548

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
44f3c40
MediaTek Flow Updates on Android Demo App Readme
cmodi-meta Oct 29, 2024
bdee603
minor changes and reordering
cmodi-meta Oct 29, 2024
38d99e0
Update mediatek_README.md
cmodi-meta Oct 29, 2024
50411c2
minor change
cmodi-meta Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 49 additions & 42 deletions examples/demo-apps/android/LlamaDemo/docs/delegates/mediatek_README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ This tutorial covers the end to end workflow for running Llama 3-8B-instruct inf
More specifically, it covers:
1. Export and quantization of Llama models against the MediaTek backend.
2. Building and linking libraries that are required to inference on-device for Android platform using MediaTek AI accelerators.
3. Loading the needed files on the device and running inference.
3. Loading the needed model files on the device and using the Android demo app to run inference.

Verified on MacOS, Linux CentOS (model export), Python 3.10, Android NDK 26.3.11579264
Phone verified: MediaTek Dimensity 9300 (D9300) chip.
Expand Down Expand Up @@ -51,19 +51,10 @@ zstd -cdq "<downloaded_buck2_file>.zst" > "<path_to_store_buck2>/buck2" && chmod
export BUCK2=path_to_buck/buck2 # Download BUCK2 and create BUCK2 executable
export ANDROID_NDK=path_to_android_ndk
export NEURON_BUFFER_ALLOCATOR_LIB=path_to_buffer_allocator/libneuron_buffer_allocator.so
export NEURON_USDK_ADAPTER_LIB=path_to_usdk_adapter/libneuronusdk_adapter.mtk.so
export ANDROID_ABIS=arm64-v8a
```

## Build Backend and MTK Llama Runner
Next we need to build and compile the MTK backend and MTK Llama runner.
```
cd examples/mediatek
./mtk_build_examples.sh
```

This will generate a cmake-android-out folder that will contain a runner executable for inferring with Llama models and another library file:
* `cmake-android-out/examples/mediatek/mtk_llama_executor_runner`
* `cmake-android-out/backends/mediatek/libneuron_backend.so`

## Export Llama Model
MTK currently supports Llama 3 exporting.

Expand Down Expand Up @@ -104,52 +95,68 @@ Note: Exporting model flow can take 2.5 hours (114GB RAM for num_chunks=4) to co

Before continuing forward, make sure to modify the tokenizer, token embedding, and model paths in the examples/mediatek/executor_runner/run_llama3_sample.sh.

## Deploy Files on Device
### Deploy
First, make sure your Android phone’s chipset version is compatible with this demo (MediaTek Dimensity 9300 (D9300)) chip. Once you have the model, tokenizer, and runner generated ready, you can push them and the .so files to the device before we start running using the runner via shell.

### Prepare to Deploy
Prior to deploying the files on device, make sure to modify the tokenizer, token embedding, and model file names in examples/mediatek/executor_runner/run_llama3_sample.sh reflect what was generated during the Export Llama Model step.
```
adb shell mkdir -p /data/local/tmp/et-mtk/ (or any other directory name)
adb push embedding_<model_name>_fp32.bin /data/local/tmp/et-mtk
adb push tokenizer.model /data/local/tmp/et-mtk
adb push <exported_prompt_model_0>.pte /data/local/tmp/et-mtk
adb push <exported_prompt_model_1>.pte /data/local/tmp/et-mtk
...
adb push <exported_prompt_model_n>.pte /data/local/tmp/et-mtk
adb push <exported_gen_model_0>.pte /data/local/tmp/et-mtk
adb push <exported_gen_model_1>.pte /data/local/tmp/et-mtk
...
adb push <exported_gen_model_n>.pte /data/local/tmp/et-mtk
```

<p align="center">
<img src="https://raw.githubusercontent.com/pytorch/executorch/refs/heads/main/docs/source/_static/img/mtk_changes_to_shell_file.png" style="width:600px">
</p>
## Populate Model Paths in Runner

In addition, create a sample_prompt.txt file with a prompt. This will be deployed to the device in the next step.
* Example content of a sample_prompt.txt file:
```
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
The Mediatek runner (`examples/mediatek/executor_runner/mtk_llama_runner.cpp`) contains the logic for implementing the function calls that come from the Android app.

You are a helpful AI assistant for travel tips and recommendations<|eot_id|><|start_header_id|>user<|end_header_id|>
**Important!** Currently the model paths are set in the runner-level. Modify the values in `examples/mediatek/executor_runner/llama_runner/llm_helper/include/llama_runner_values.h` to set the model paths, tokenizer path, embedding file path, and other metadata.

What can you help me with?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```

### Deploy
First, make sure your Android phone’s chipset version is compatible with this demo (MediaTek Dimensity 9300 (D9300)) chip. Once you have the model, tokenizer, and runner generated ready, you can push them and the .so files to the device before we start running using the runner via shell.
## Build AAR Library

Next we need to build and compile the MediaTek backend and MediaTek Llama runner. By setting `NEURON_BUFFER_ALLOCATOR_LIB`, the script will build the MediaTek backend.
```
adb shell mkdir -p /data/local/tmp/llama
adb push examples/mediatek/executor_runner/run_llama3_sample.sh /data/local/tmp/llama
adb push sample_prompt.txt /data/local/tmp/llama
adb push cmake-android-out/examples/mediatek/mtk_llama_executor_runner /data/local/tmp/llama
adb push cmake-android-out/backends/mediatek/libneuron_backend.so /data/local/tmp/llama
adb push libneuron_buffer_allocator.so /data/local/tmp/llama
adb push libneuronusdk_adapter.mtk.so /data/local/tmp/llama
adb push embedding_<model_name>_fp32.bin /data/local/tmp/llama
adb push tokenizer.model /data/local/tmp/llama
sh build/build_android_llm_demo.sh
```

**Output**: This will generate an .aar file that is already imported into the expected directory for the Android app. It will live in `examples/demo-apps/android/Llamademo/app/libs`.

If you were to unzip the .aar file or open it in Android Studio, verify it contains the following related to MediaTek backend:
* libneuron_buffer_allocator.so
* libneuronusdk_adapter.mtk.so
* libneuron_backend.so (generated during build)

## Run Demo
At this point we have pushed all the required files on the device and we are ready to run the demo!
```
adb shell

<android_device>:/ $ cd data/local/tmp/llama
<android_device>:/data/local/tmp/llama $ sh run_llama3_sample.sh
### Alternative 1: Android Studio (Recommended)
1. Open Android Studio and select “Open an existing Android Studio project” to open examples/demo-apps/android/LlamaDemo.
2. Run the app (^R). This builds and launches the app on the phone.

### Alternative 2: Command line
Without Android Studio UI, we can run gradle directly to build the app. We need to set up the Android SDK path and invoke gradle.
```
export ANDROID_HOME=<path_to_android_sdk_home>
pushd examples/demo-apps/android/LlamaDemo
./gradlew :app:installDebug
popd
```
If the app successfully run on your device, you should see something like below:

<p align="center">
<img src="https://raw.githubusercontent.com/pytorch/executorch/refs/heads/main/docs/source/_static/img/mtk_output.png" style="width:800px">
<img src="https://raw.githubusercontent.com/pytorch/executorch/refs/heads/main/docs/source/_static/img/opening_the_app_details.png" style="width:800px">
</p>

Once you've loaded the app on the device:
1. Click on the Settings in the app
2. Select MediaTek from the Backend dropdown
3. Click the "Load Model" button. This will load the models from the Runner

## Reporting Issues
If you encountered any bugs or issues following this tutorial please file a bug/issue here on [Github](https://github.com/pytorch/executorch/issues/new).
Loading