Merge pull request #1331 from mitsunami/koki/android-onnxruntime

Build an Android chat app with ONNX Runtime Learning Path

Author: pareenaverma

content/learning-paths/smartphones-and-mobile/build-android-chat-app-using-onnxruntime/1-dev-env-setup.md

@@ -0,0 +1,63 @@
+---
+title: Create a development environment
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Set up your development environment
+
+In this Learning Path, you will learn how to build and deploy a simple LLM-based chat app to an Android device using ONNX Runtime. You will learn how to build the ONNX runtime and ONNX Runtime generate() API and how to run the Phi-3 model for the Android application.
+
+The first step is to prepare a development environment with the required software:
+
+- Android Studio (latest version recommended)
+- Android NDK (tested with version 27.0.12077973)
+- Python 3.11
+- CMake (tested with version 3.28.1)
+- Ninja (tested with version 1.11.1)
+
+The instructions were tested on an x86 Windows machine with at least 16GB of RAM.
+
+## Install Android Studio and Android NDK
+
+Follow these steps to install and configure Android Studio:
+
+1. Download and install the latest version of [Android Studio](https://developer.android.com/studio/). 
+
+2. Navigate to `Tools -> SDK Manager`.
+
+3. In the `SDK Platforms` tab, check `Android 14.0 ("UpsideDownCake")`.
+
+4. In the `SDK Tools` tab, check `NDK (Side by side)`.
+
+5. Click Ok and Apply.
+
+## Install Python 3.11
+
+Download and install [Python version 3.11](https://www.python.org/downloads/release/python-3110/)
+
+## Install CMake
+
+CMake is an open-source tool that automates the build process for software projects, helping to generate platform-specific build configurations.
+
+[Download and install CMake](https://cmake.org/download/)
+
+{{% notice Note %}}
+The instructions were tested with version 3.28.1
+{{% /notice %}}
+
+## Install Ninja
+
+Ninja is a minimalistic build system designed to efficiently handle incremental builds, particularly in large-scale software projects, by focusing on speed and simplicity.
+
+The Ninja generator needs to be used to build on Windows for Android.
+
+[Download and install Ninja]( https://github.com/ninja-build/ninja/releases)
+
+{{% notice Note %}}
+The instructions were tested with version 1.11.1
+{{% /notice %}}
+
+You now have the required development tools installed to follow this learning path.

content/learning-paths/smartphones-and-mobile/build-android-chat-app-using-onnxruntime/2-build-onnxruntime.md

@@ -0,0 +1,57 @@
+---
+title: Build ONNX Runtime
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Cross-compile ONNX Runtime for Android CPU
+
+Now that you have your environment set up correctly, you can build the ONNX Runtime inference engine. 
+
+ONNX Runtime is an open-source inference engine designed to accelerate the deployment of machine learning models, particularly those in the Open Neural Network Exchange (ONNX) format. ONNX Runtime is optimized for high performance and low latency, making it popular for production deployment of AI models. You can learn more by reading the [ONNX Runtime Overview](https://onnxruntime.ai/).
+
+
+### Clone onnxruntime repo
+
+Open up a Windows Powershell and checkout the source tree:
+
+```bash
+cd C:\Users\$env:USERNAME
+git clone --recursive https://github.com/Microsoft/onnxruntime.git
+cd onnxruntime
+git checkout 9b37b3ea4467b3aab9110e0d259d0cf27478697d
+```
+
+{{% notice Note %}}
+You might be able to use a later commit. These steps have been tested with the commit `9b37b3ea4467b3aab9110e0d259d0cf27478697d`.
+{{% /notice %}}
+
+### Build for Android CPU
+
+The Ninja generator needs to be used to build on Windows. First, set JAVA_HOME to the path to your JDK install. You can point to the JDK from Android Studio, or a standalone JDK install.
+
+```bash
+$env:JAVA_HOME="C:\Program Files\Android\Android Studio\jbr"
+```
+
+Now run the following command:
+
+```bash
+
+./build.bat --config Release --build_shared_lib --android --android_sdk_path C:\Users\$env:USERNAME\AppData\Local\Android\Sdk --android_ndk_path C:\Users\$env:USERNAME\AppData\Local\Android\Sdk\ndk\27.0.12077973 --android_abi arm64-v8a --android_api 27 --cmake_generator Ninja --build_java
+
+```
+
+Android Archive (AAR) files, which can be imported directly in Android Studio, will be generated by using the above command with `--build_java`
+
+When the build is complete, confirm the shared library and the AAR file have been created:
+
+```
+ls build\Windows\Release\onnxruntime.so
+ls build\Windows\Release\java\build\android\outputs\aar\onnxruntime-release.aar
+```
+
+
+

content/learning-paths/smartphones-and-mobile/build-android-chat-app-using-onnxruntime/3-build-onnxruntime-generate-api.md

@@ -0,0 +1,41 @@
+---
+title: Build ONNX Runtime Generate() API
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Cross-compile the ONNX Runtime generate() API for Android CPU
+
+The Generate() API in ONNX Runtime is designed for text generation tasks using models like Phi-3. It implements the generative AI loop for ONNX models, including pre and post processing, inference with ONNX Runtime, logits processing, search and sampling, and KV cache management. You can learn more by reading the [ONNX Runtime generate() API page](https://onnxruntime.ai/docs/genai/).
+
+
+### Clone onnxruntime-genai repo
+Within your Windows Powershell prompt, checkout the source repo:
+
+```bash
+C:\Users\$env:USERNAME
+git clone https://github.com/microsoft/onnxruntime-genai
+cd onnxruntime-genai
+git checkout 1e4d289502a61265c3b07efb17d8796225bb0b7f
+```
+
+{{% notice Note %}}
+You might be able to use later commits. These steps have been tested with the commit `1e4d289502a61265c3b07efb17d8796225bb0b7f`.
+{{% /notice %}}
+
+### Build for Android CPU
+
+The Ninja generator needs to be used to build on Windows for Android. Make sure JAVA_HOME is set before running the following command:
+
+```bash
+python -m pip install requests 
+python3.11 build.py --build_java --android --android_home C:\Users\$env:USERNAME\AppData\Local\Android\Sdk --android_ndk_path C:\Users\$env:USERNAME\AppData\Local\Android\Sdk\ndk\27.0.12077973 --android_abi arm64-v8a --config Release
+```
+
+When the build is complete, confirm the shared library has been created:
+
+```output
+ls build\Android\Release\onnxruntime-genai.so
+```

content/learning-paths/smartphones-and-mobile/build-android-chat-app-using-onnxruntime/4-run-benchmark-on-android.md

@@ -0,0 +1,88 @@
+---
+title: Run Benchmark on Android phone
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Run example code for running Phi-3-mini
+
+
+### Build model runner
+
+You will now cross-compile the model runner to run on Android using the commands below:
+
+``` bash
+cd onnxruntime-genai
+copy src\ort_genai.h examples\c\include\
+copy src\ort_genai_c.h examples\c\include\
+cd examples\c
+mkdir build
+cd build
+```
+Run the cmake command as shown:
+
+```bash
+cmake -DCMAKE_TOOLCHAIN_FILE=C:\Users\$env:USERNAME\AppData\Local\Android\Sdk\ndk\27.0.12077973\build\cmake\android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-27 -DCMAKE_BUILD_TYPE=Release -G "Ninja" ..
+ninja
+```
+
+After successful build, a binary program called `phi3` will be created.
+
+### Prepare phi-3-mini model
+
+Phi-3 ONNX models are hosted on HuggingFace. You can download the Phi-3-mini model with huggingface-cli command:
+
+``` bash
+pip install huggingface-hub[cli]
+huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir .
+```
+This command downloads the model into a folder called cpu_and_mobile.
+
+The phi-3-mini (3B) model has a short (4k) context version and a long (128k) context version. The long context version can accept much longer prompts and produce longer output text, but it does consume more memory. In this learning path, you will use the short context version, which is quantized to 4-bits.
+
+
+### Run on Android via adb shell
+
+#### Connect your android phone
+Connect your phone to your computer using a USB cable. 
+
+You need to enable USB debugging on your Android device. You can follow [Configure on-device developer options](https://developer.android.com/studio/debug/dev-options) to enable USB debugging.
+
+Once you have enabled USB debugging and connected via USB, run:
+
+```
+adb devices
+```
+
+You should see your device listed to confirm it is connected. 
+
+#### Copy the runner binary and the model files to the phone
+
+``` bash
+adb push cpu-int4-rtn-block-32-acc-level-4 /data/local/tmp
+adb push .\phi3 /data/local/tmp
+adb push onnxruntime-genai\build\Android\Release\libonnxruntime-genai.so /data/local/tmp
+adb push onnxruntime\build\Windows\Release\libonnxruntime.so /data/local/tmp
+```
+
+#### Run the model
+
+Use the runner to execute the model on the phone with the `adb` command:
+
+``` bash
+adb shell
+cd /data/local/tmp
+chmod 777 phi3
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data/local/tmp
+./phi3 cpu-int4-rtn-block-32-acc-level-4
+```
+
+This will allow the runner program to load the model, and then it will prompt you to input the text prompt to the model. After you enter yout input prompt, the text output by the model will be displayed. On completion, the performance metrics similar to what is shown below should be displayed:
+
+```
+Prompt length: 64, New tokens: 931, Time to first: 1.79s, Prompt tokens per second: 35.74 tps, New tokens per second: 6.34 tps
+```
+
+You have successfully run the Phi-3 model on your Android smartphone powered by Arm.

content/learning-paths/smartphones-and-mobile/build-android-chat-app-using-onnxruntime/5-build-android-chat-app.md

@@ -0,0 +1,54 @@
+---
+title: Build and Run Android chat app
+weight: 6
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Build Android chat app
+
+Another way to run the model is to use an Android GUI app.
+You can use the Android demo application included in the [onnxruntime-inference-examples repository](https://github.com/microsoft/onnxruntime-inference-examples) to demonstrate local inference.
+
+### Clone the repo
+
+``` bash
+git clone https://github.com/microsoft/onnxruntime-inference-examples
+cd onnxruntime-inference-examples
+git checkout 009920df0136d7dfa53944d06af01002fb63e2f5
+```
+
+{{% notice Note %}}
+You could probably use a later commit but these steps have been tested with the commit `009920df0136d7dfa53944d06af01002fb63e2f5`.
+{{% /notice %}}
+
+### Build the app using Android Studio
+
+Open the `mobile\examples\phi-3\android` directory with Android Studio.
+
+#### (Optional) In case you want to use ONNX Runtime AAR you built
+
+Copy ONNX Runtime AAR you built before if needed:
+
+```bash
+Copy onnxruntime\build\Windows\Release\java\build\android\outputs\aar\onnxruntime-release.aar mobile\examples\phi-3\android\app\libs
+```
+
+Update `build.gradle.kts (:app)` as below:
+
+``` kotlin
+// ONNX Runtime with GenAI
+//implementation("com.microsoft.onnxruntime:onnxruntime-android:latest.release")
+implementation(files("libs/onnxruntime-release.aar"))
+```
+
+After that, click `File`->`Sync Project with Gradle`
+
+#### Build and run the app
+
+When you press Run, the build will be executed, and then the app will be copied and installed on the Android device. This app will automatically download the Phi-3-mini model during the first run. After the download, you can input the prompt in the text box and execute it to run the model.
+
+You should now see a running app on your phone that looks like this:
+
+![App screenshot](screenshot.png)

content/learning-paths/smartphones-and-mobile/build-android-chat-app-using-onnxruntime/_index.md

@@ -0,0 +1,40 @@
+---
+title: Build an Android chat application with ONNX Runtime API
+
+minutes_to_complete: 60
+
+who_is_this_for: This is an advanced topic for software developers interested in learning how to build an Android chat app with ONNX Runtime and ONNX Runtime Generate() API.
+
+learning_objectives: 
+    - Build ONNX Runtime and ONNX Runtime generate() API for Android.
+    - Run the Phi-3 model using ONNX Runtime on an Arm-based smartphone.
+
+prerequisites:
+    - A Windows x86_64 development machine with at least 16GB of RAM. You should also be able to use Linux or MacOS for the build, but the instructions for it have not been included in this learning path.
+    - An Android phone with at least 8GB of RAM. This learning path was tested on Samsung Galaxy S24.
+
+author_primary: Koki Mitsunami
+
+### Tags
+skilllevels: Advanced
+subjects: ML
+armips:
+    - Cortex-A
+    - Cortex-X
+tools_software_languages:
+    - Kotlin
+    - C++
+    - ONNX Runtime
+    - Android
+    - Mobile
+operatingsystems:
+    - Windows
+    - Android
+
+
+### FIXED, DO NOT MODIFY
+# ================================================================================
+weight: 1                       # _index.md always has weight of 1 to order correctly
+layout: "learningpathall"       # All files under learning paths have this same wrapper
+learning_path_main_page: "yes"  # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
+---

content/learning-paths/smartphones-and-mobile/build-android-chat-app-using-onnxruntime/_next-steps.md

@@ -0,0 +1,27 @@
+---
+next_step_guidance: Now that you are familiar with building LLM applications with ONNX Runtime, you are ready to incorporate LLMs into your Android applications. You can learn how to further accelerate the performance of your LLMs using KleidiAI.
+
+recommended_path: /learning-paths/cross-platform/kleidiai-explainer/
+
+further_reading:
+    - resource:
+        title: ONNX Runtime
+        link: https://onnxruntime.ai/docs/
+        type: documentation
+    - resource:
+        title: ONNX Runtime generate() API
+        link: https://onnxruntime.ai/docs/genai/
+        type: documentation
+    - resource:
+        title: Accelerating AI Developer Innovation Everywhere with New Arm Kleidi
+        link: https://newsroom.arm.com/blog/arm-kleidi
+        type: blog
+
+
+# ================================================================================
+#       FIXED, DO NOT MODIFY
+# ================================================================================
+weight: 21                  # set to always be larger than the content in this path, and one more than 'review'
+title: "Next Steps"         # Always the same
+layout: "learningpathall"   # All files under learning paths have this same wrapper
+---