Merge pull request #1732 from annietllnd/vision-llm

Technical review of Vision LLM LP

Author: pareenaverma

assets/contributors.csv

@@ -47,7 +47,7 @@ Alaaeddine Chakroun,Day Devs,Alaaeddine-Chakroun,alaaeddine-chakroun,,https://da
 Koki Mitsunami,Arm,,kmitsunami,,
 Chen Zhang,Zilliz,,,,
 Tianyu Li,Arm,,,,
-Georgios Mermigkis,VectorCamp,gMerm,georgios-mermigkis,,https://vectorcamp.gr/ 
+Georgios Mermigkis,VectorCamp,gMerm,georgios-mermigkis,,https://vectorcamp.gr/
 Ben Clark,Arm,,,,
 Han Yin,Arm,hanyin-arm,nacosiren,,
 Willen Yang,Arm,,,,
@@ -80,3 +80,5 @@ Tom Pilar,,,,,
 Cyril Rohr,,,,,
 Odin Shen,Arm,odincodeshen,odin-shen-lmshen,,
 Avin Zarlez,Arm,AvinZarlez,avinzarlez,,https://www.avinzarlez.com/
+Shuheng Deng,Arm,,,,
+Yiyang Fan,Arm,,,,

content/learning-paths/mobile-graphics-and-gaming/Vision-LLM-inference-on-Android-with-KleidiAI-and-MNN/1-devenv-and-model.md

@@ -0,0 +1,90 @@
+---
+title: Build the MNN Android Demo with GUI
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+In this section, you will set up a development environment by installing dependencies and preparing the Qwen vision model.
+
+## Install required software
+
+Install the Android NDK (Native Development Kit) and git-lfs. This learning path was tested with NDK version `28.0.12916984` and CMake version `4.0.0-rc1`.
+
+For Ubuntu or Debian systems, you can install CMake and git-lfs with the following command:
+
+```bash
+sudo apt update
+sudo apt install cmake git-lfs -y
+```
+
+You can use Android Studio to obtain the NDK. Click **Tools > SDK Manager**, and navigate to the the SDK Tools tab. Select the NDK (Side by side) and CMake checkboxes, as shown below:
+
+![Install NDK](./install_ndk.png)
+
+Refer to [Install NDK and CMake](https://developer.android.com/studio/projects/install-ndk) for other installation methods.
+
+Make sure Python and pip is installed by verifying a version is printed on running this command:
+
+```bash
+python --version
+pip --version
+```
+
+{{% notice Note %}}
+The above commands may fail when Python is installed if Python 3.x is not the default version. You can try running `python3 --version` and `pip3 --version` to be sure.
+{{% /notice %}}
+
+## Set up phone connection
+
+You will need to set up an authorized connection with your phone. The Android SDK Platform Tools package, included in Android Studio, comes with Android Debug Bridge (ADB). You will use this tool to transfer files later on.
+
+Connect your phone to the computer using a USB cable. You will need to activate USB debugging on your phone. Find the **Build Number** in your **Settings** app and tap it 7 times. Then, enable **USB debugging** in **Developer Options**.
+
+You should now see your device listed upon running `adb devices`:
+
+```output
+List of devices attached
+<DEVICE ID>     device
+```
+
+## Download and convert the model
+
+The following commands download the model from Hugging Face, and clones a tool for exporting LLM model to the MNN framework.
+
+```bash
+cd $HOME
+pip install -U huggingface_hub
+huggingface-cli download Qwen/Qwen2-VL-2B-Instruct --local-dir ./Qwen2-VL-2B-Instruct/
+git clone https://github.com/wangzhaode/llm-export
+cd llm-export && pip install .
+```
+
+You can use the `llm-export` repository to quantize the model with the following options:
+
+```bash
+llmexport --path ../Qwen2-VL-2B-Instruct/ --export mnn --quant_bit 4 \
+    --quant_block 0 --dst_path Qwen2-VL-2B-Instruct-convert-4bit-per_channel --sym
+```
+
+The table below gives you an explanation of the different arguments:
+
+| Parameter        | Description | Explanation |
+|------------------|-------------|--------------|
+| `--quant_bit` | mnn quant bit, 4 or 8, default is 4 | `4` represents q4 quantization. |
+| `--quant_block` | mnn quant block, default is 0 | `0` represents per-channel quantization, `128` represents 128 per-block quantization. |
+| `--sym` | symmetric quantization (without zeropoint), defualt is False. | The quantization parameter that enables symmetrical quantization. |
+
+To learn more about the parameters, refer to the [transformers README.md](https://github.com/alibaba/MNN/tree/master/transformers).
+
+Verify the model is built correct by checking the size of the resulting model. The `Qwen2-VL-2B-Instruct-convert-4bit-per_channel` directory should be at least 1 GB in size.
+
+Push the model onto the device:
+
+```shell
+adb shell mkdir /data/local/tmp/models/
+adb push Qwen2-VL-2B-Instruct-convert-4bit-per_channel /data/local/tmp/models
+```
+
+With the model set up, it's time to use Android Studio to build and run an example application.

content/learning-paths/mobile-graphics-and-gaming/Vision-LLM-inference-on-Android-with-KleidiAI-and-MNN/2-generate-apk.md

@@ -0,0 +1,48 @@
+---
+title: Benchmark the Vision Transformer performance with KleidiAI
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+In this section, you will try the Qwen model in action using a demo application using a Android Package Kit (APK)
+
+## Clone MNN repo
+
+A fork of the upstream MNN repository is set up to enable building the app as an Android Studio project. Run the following to clone the repository and checkout the source tree:
+
+```bash
+cd $HOME
+git clone https://github.com/HenryDen/MNN.git
+cd MNN
+git checkout origin/llm_android_demo
+```
+
+## Build the app using Android Studio
+
+### Open project and build
+
+Open Android Studio. Go to **File > Open**. Navigate to the MNN repository you just cloned. Expand the `transformers/llm/engine/` directories, select the `android` one and click `Open`.
+
+This will trigger a build of the project, and you should see a similar output on completion:
+
+```output
+BUILD SUCCESSFUL in 1m 42s
+```
+
+### Generate and run the APK
+
+Navigate to **Build > Generate App Bundles or APKs**. Select **Generate APKs**.
+
+The build will be executed, and then the app will be copied and installed on the Android device.
+
+After opening the app, you will see the splash screen:
+
+![Loading screenshot](Loading_page.png)
+
+Finally, you can use the UI to chat with the app. Try uploading an image and ask a question on it.
+
+![Loading screenshot](chat2.png)
+
+The final step is to examine how KleidiAI can improve the performance of the model. Continue to the next section to find out.

content/learning-paths/mobile-graphics-and-gaming/Vision-LLM-inference-on-Android-with-KleidiAI-and-MNN/3-benchmark.md

@@ -0,0 +1,140 @@
+---
+title: Build the MNN Command-line ViT Demo
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+In this section, you will use the model to benchmark performance with and without KleidiAI kernels. You will need to compile library files to run the optimized inference.
+
+## Prepare an example image
+
+You will use an image to run a command-line prompt. In this learning path, the tiger below will be used as an example. You can save this image or provide one of your own. Re-name the image to `example.png` in order to use the commands in the following sections.
+
+![example image](example.png)
+
+Use ADB to load the image onto your phone:
+
+```bash
+adb push example.png /data/local/tmp/
+```
+
+## Build binaries for command-line inference
+
+Navigate to the MNN project you cloned in the previous section. Create a build directory and run the script. The first time, you will build the binaries with the `-DMNN_KLEIDIAI` flag set to `FALSE`.
+
+```bash
+cd $HOME/MNN/project/android
+mkdir build_64 && cd build_64
+
+../build_64.sh "-DMNN_LOW_MEMORY=true -DLLM_SUPPORT_VISION=true -DMNN_KLEIDIAI=FALSE  \
+  -DMNN_CPU_WEIGHT_DEQUANT_GEMM=true -DMNN_BUILD_LLM=true \
+  -DMNN_SUPPORT_TRANSFORMER_FUSE=true -DMNN_ARM82=true -DMNN_OPENCL=true \
+  -DMNN_USE_LOGCAT=true -DMNN_IMGCODECS=true -DMNN_BUILD_OPENCV=true"
+```
+{{% notice Note %}}
+If your NDK toolchain isn't set up correctly, you may run into issues with the above script. Make note of where the NDK was installed - this will be a directory named after the version you downloaded earlier. Try exporting the following environment variables before re-running `build_64.sh`.
+
+```bash
+export ANDROID_NDK_HOME=<path-to>/ndk/28.0.12916984
+
+export CMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake
+export ANDROID_NDK=$ANDROID_NDK_HOME
+```
+{{% /notice %}}
+
+Push the files to your mobile device. Then, enter a shell on the phone using ADB.
+
+```bash
+adb push *so llm_demo tools/cv/*so /data/local/tmp/
+adb shell
+```
+
+The following commands should be run in the ADB shell. Navigate to the directory you pushed the files to, add executable permissions to the `llm_demo` file and export an environment variable for it to run properly. After this, use the example image you transferred earlier to create a file containing the text content for the prompt.
+
+```bash
+cd /data/local/tmp/
+chmod +x llm_demo
+export LD_LIBRARY_PATH=$PWD
+echo "<img>./example.png</img>Describe the content of the image." > prompt
+```
+
+Finally, run an inference on the model with the following command.
+
+```bash
+./llm_demo models/Qwen-VL-2B-convert-4bit-per_channel/config.json prompt
+```
+
+If the launch is successful, you should see the following output, with the performance benchmark at the end.
+
+```output
+config path is models/Qwen-VL-2B-convert-4bit-per_channel/config.json
+tokenizer_type = 3
+prompt file is prompt
+The image features a tiger standing in a grassy field, with its front paws raised and its eyes fixed on something or someone behind it. The tiger's stripes are clearly visible against the golden-brown background of the grass. The tiger appears to be alert and ready for action, possibly indicating a moment of tension or anticipation in the scene.
+
+#################################
+prompt tokens num = 243
+decode tokens num = 70
+ vision time = 5.76 s
+  audio time = 0.00 s
+prefill time = 1.26 s
+ decode time = 2.02 s
+prefill speed = 192.28 tok/s
+ decode speed = 34.73 tok/s
+##################################
+```
+
+## Enable KleidiAI and re-run inference
+
+The next step is to re-generate the binaries with KleidiAI activated. This is done by updating the flag `-DMNN_KLEIDIAI` to `TRUE`. From the `build_64` directory, run:
+```bash
+../build_64.sh "-DMNN_LOW_MEMORY=true -DLLM_SUPPORT_VISION=true -DMNN_KLEIDIAI=TRUE \
+-DMNN_CPU_WEIGHT_DEQUANT_GEMM=true -DMNN_BUILD_LLM=true \
+-DMNN_SUPPORT_TRANSFORMER_FUSE=true -DMNN_ARM82=true -DMNN_OPENCL=true \
+-DMNN_USE_LOGCAT=true -DMNN_IMGCODECS=true -DMNN_BUILD_OPENCV=true"
+```
+
+The next step is to update the files on your phone. Start by removing the ones used in the previous step. Then, push the new ones with the same command as before.
+
+```bash
+adb shell "cd /data/local/tmp; rm -rf *so llm_demo tools/cv/*so"
+adb push *so llm_demo tools/cv/*so /data/local/tmp/
+adb shell
+```
+
+In the new ADB shell, preform the same steps as in the previous section.
+
+```bash
+cd /data/local/tmp/
+chmod +x llm_demo
+export LD_LIBRARY_PATH=$PWD
+./llm_demo models/Qwen-VL-2B-convert-4bit-per_channel/config.json prompt
+```
+
+The same output should be displayed, with the benchmark printed at the end:
+```output
+#################################
+prompt tokens num = 243
+decode tokens num = 70
+ vision time = 2.91 s
+  audio time = 0.00 s
+prefill time = 0.91 s
+ decode time = 1.56 s
+prefill speed = 266.13 tok/s
+ decode speed = 44.96 tok/s
+##################################
+```
+
+This time, you should see an improvement in the benchmark. Below is an example table showing the uplift on three relevant metrics after enabling the KleidiAI kernels.
+
+| Benchmark           | Without KleidiAI | With KleidiAI |
+|---------------------|------------------|---------------|
+| Vision Process Time | 5.76 s           | 2.91 s        |
+| Prefill Speed       | 192.28 tok/s     | 266.13 tok/s  |
+| Decode Speed        | 34.73 tok/s      | 44.96 tok/s   |
+
+The prefill speed describes how fast the model processes the input prompt. The decode speed corresponds to the rate at which the model generates new tokens after the input is processed
+
+This shows the advantages of using Arm optimized kernels for your ViT use-cases.