Skip to content

Commit abbd721

Browse files
Vinayak Katochekanshibu
Vinayak Katoch
authored and
ekanshibu
committed
Add qnn_run.md for QNN Model Running Instructions
This commit introduces the `qnn_run.md` document, which provides detailed instructions for setting up and running QNN models on Linux hosts and target devices. The document covers installing the QNN SDK, setting up dependencies, converting CNN models to QNN format, building models, and preparing and running the models on target devices. Signed-off-by: Vinayak Katoch <vkatoch@hu-vkatoch-hyd.qualcomm.com>
1 parent e037ab8 commit abbd721

File tree

1 file changed

+311
-0
lines changed

1 file changed

+311
-0
lines changed

‎test/qnn_run.md‎

Lines changed: 311 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,311 @@
1+
# Document for Running QNN Models
2+
3+
## For Linux Host (Setup)
4+
5+
### Step 1: Install Qualcomm AI Engine Direct (QNN SDK)
6+
7+
1. Download the QNN SDK:
8+
* Go to the QNN SDK product [page](https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct-sdk).
9+
* Click **Get Software** to download the QNN SDK.
10+
* Unzip the downloaded SDK.
11+
2. Set Up Your Environment:
12+
* Open a terminal.
13+
* Navigate to `qairt/<QNN_SDK_VERSION>` inside the unzipped QNN SDK.
14+
* Run the following commands to set up the environment:
15+
+ `cd bin`
16+
+ `source ./envsetup.sh`
17+
* This will automatically set the environment variables `QNN_SDK_ROOT` and `SNPE_ROOT`.
18+
* To save this for future bash terminals, run:
19+
+ `echo 'export QNN_SDK_ROOT="${QNN_SDK_ROOT}"' >> ~/.bashrc`
20+
3. Check Dependencies:
21+
* Run:
22+
+ `sudo ${QNN_SDK_ROOT}/bin/check-linux-dependency.sh`
23+
* When all necessary dependencies are installed, the script will display “Done!!”.
24+
4. Verify Toolchain Installation:
25+
* Run:
26+
+ `${QNN_SDK_ROOT}/bin/envcheck -c`
27+
* This will verify that the required toolchain is installed successfully.
28+
29+
### Step 2: Install QNN SDK Dependencies
30+
31+
1. Install Python 3.10:
32+
* Run the following commands:
33+
+ `sudo apt-get update && sudo apt-get install python3.10 python3-distutils libpython3.10`
34+
* Verify the installation by running:
35+
+ `python3 --version`
36+
2. Navigate to QNN SDK Root:
37+
* Run:
38+
+ `cd ${QNN_SDK_ROOT}`
39+
3. Create and Activate a Virtual Environment:
40+
* Run the following commands (you may rename `MY_ENV_NAME` to any name you prefer):
41+
+ `python3 -m venv MY_ENV_NAME`
42+
+ `source MY_ENV_NAME/bin/activate`
43+
4. Update Dependencies:
44+
* Run:
45+
+ `python "${QNN_SDK_ROOT}/bin/check-python-dependency"`
46+
47+
### Step 3: Install Model Frameworks
48+
49+
1. Identify Relevant Frameworks:
50+
* QNN supports multiple model frameworks. Install only the ones relevant for the AI model files you want to use.
51+
* **Warning:** You do not need to install all packages listed here.
52+
2. Note:
53+
* You can install a package by running:
54+
+ `pip3 install package==version`
55+
* Example Installations:
56+
+ `pip3 install torch==1.13.1`
57+
+ `pip3 install tensorflow==2.10.1`
58+
+ `pip3 install tflite==2.3.0`
59+
+ `pip3 install torchvision==0.14.1`
60+
+ `pip3 install onnx==1.12.0`
61+
+ `pip3 install onnxruntime==1.17.1`
62+
+ `pip3 install onnxsim==0.4.36`
63+
64+
### Step 4: Install Target Device OS-Specific Toolchain Code
65+
66+
#### Working with an Android Target Device:
67+
68+
1. Install Android NDK:
69+
* Download the [Android NDK r26c](https://dl.google.com/android/repository/android-ndk-r26c-linux.zip).
70+
* Unzip the file.
71+
2. Set Up Environment Variables:
72+
* Open a terminal.
73+
* Replace `<path-to-your-android-ndk-root-folder>` with the path to the unzipped `android-ndk-r26c` folder, then run:
74+
+ `echo 'export ANDROID_NDK_ROOT="<path-to-your-android-ndk-root-folder>"' >> ~/.bashrc`
75+
* Add the location of the unzipped file to your PATH by running:
76+
+ `echo 'export PATH="${ANDROID_NDK_ROOT}:${PATH}"' >> ~/.bashrc`
77+
+ `source ~/.bashrc`
78+
3. Verify Environment Variables:
79+
* Run:
80+
+ `${QNN_SDK_ROOT}/bin/envcheck -n`
81+
82+
#### Working with a Linux Target Device:
83+
84+
1. Verify Clang++14 Installation:
85+
* For Linux target devices, you will likely need to use clang++14 to build models for them using the QNN SDK.
86+
* You can verify if you have clang++14 by running:
87+
+ `${QNN_SDK_ROOT}/bin/envcheck -c`
88+
2. Set Up Cross Compiler (for **Linux Embedded**):
89+
* Open a new terminal
90+
* Follow these steps to set up the cross compiler:
91+
+ `wget https://artifacts.codelinaro.org/artifactory/qli-ci/flashable-binaries/qimpsdk/qcm6490/x86/qcom-6.6.28-QLI.1.1-Ver.1.1_qim-product-sdk-1.1.3.zip`
92+
+ `unzip qcom-6.6.28-QLI.1.1-Ver.1.1_qim-product-sdk-1.1.3.zip`
93+
+ `umask a+rx`
94+
+ `cd /path/to/unzip/target/qcm6490/sdk`
95+
+ `sh qcom-wayland-x86_64-qcom-multimedia-image-armv8-2a-qcm6490-toolchain-ext-1.0.sh`
96+
+ `export ESDK_ROOT=<path of installation directory>`
97+
+ `cd $ESDK_ROOT`
98+
+ `source environment-setup-armv8-2a-qcom-linux`
99+
100+
## CNN to QNN for Linux Host
101+
102+
### Step 1: Set Up The Example TensorFlow Model
103+
104+
1. Verify TensorFlow Installation:
105+
* Run:
106+
+ `python3 -m pip show tensorflow`
107+
2. Set the TENSORFLOW_HOME Environment Variable:
108+
* Run the following commands:
109+
```
110+
tensorflowLocation=$(python -m pip show tensorflow | grep '^Location: ' | awk '{print $2}')
111+
export TENSORFLOW_HOME="$tensorflowLocation/tensorflow"
112+
echo "export TENSORFLOW_HOME=$tensorflowLocation/tensorflow" >> ~/.bashrc
113+
```
114+
3. Verify Environment Variable:
115+
* Run:
116+
+ `${TENSORFLOW_HOME}`
117+
4. Create Test Data:
118+
* Run:
119+
+ `python3 ${QNN_SDK_ROOT}/examples/Models/InceptionV3/scripts/setup_inceptionv3.py -a ~/tmpdir -d`
120+
* This will create the test data for this tutorial:
121+
+ Model file at: `${QNN_SDK_ROOT}/examples/Models/InceptionV3/tensorflow/inception_v3_2016_08_28_frozen.pb`
122+
+ Raw images at: `${QNN_SDK_ROOT}/examples/Models/InceptionV3/data/cropped`
123+
124+
### Step 2: Convert the CNN Model into a QNN Model
125+
126+
1. Quantized Model Conversion:
127+
* To use a quantized model, pass in the `--input_list` flag to specify the input.
128+
* Run the following command:
129+
```
130+
python ${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-tensorflow-converter \
131+
--input_network "${QNN_SDK_ROOT}/examples/Models/InceptionV3/tensorflow/inception_v3_2016_08_28_frozen.pb" \
132+
--input_dim input 1,299,299,3 \
133+
--input_list "${QNN_SDK_ROOT}/examples/Models/InceptionV3/data/cropped/raw_list.txt" \
134+
--out_node "InceptionV3/Predictions/Reshape_1" \
135+
--output_path "${QNN_SDK_ROOT}/examples/Models/InceptionV3/model/Inception_v3.cpp"
136+
```
137+
2. Flags Explanation:
138+
* `--input_network`: Path to the source framework model.
139+
* `--input_dim`: Input name followed by dimensions.
140+
* `--input_list`: Absolute path to input data (required for quantization).
141+
* `--out_node`: Name of the graph’s output Tensor Names.
142+
* `--output_path`: Indicates where to put the output files.
143+
3. Artifacts Produced:
144+
* `Inception_v3.cpp`: Contains the sequence of API calls.
145+
* `Inception_v3.bin`: Static data associated with the model.
146+
* `Inception_v3_net.json`: Describes the model data in a standardized manner (not needed to build the QNN model later on).
147+
4. Location of Artifacts:
148+
* You can find the artifacts in the `${QNN_SDK_ROOT}/examples/Models/InceptionV3/model` directory.
149+
150+
### Other Model Types:
151+
If you are using another type of model, you can refer to the [Tools page](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/tools.html) for a table of potential scripts to help convert them into QNN format. These scripts will have a similar qnn-model-type-converter naming convention.
152+
153+
### Step 3: Model Build on Linux Host
154+
155+
1. Choose Target Architecture:
156+
* Select the most relevant supported target architecture from the following list:
157+
+ 64-bit Linux targets: `x86_64-linux-clang`
158+
+ 64-bit Android devices: `aarch64-android`
159+
+ Qualcomm’s QNX OS: `aarch64-qnx` (Note: This architecture is not supported by default in the QNN SDK.)
160+
+ OpenEmbedded Linux (GCC 11.2): `aarch64-oe-linux-gcc11.2`
161+
+ OpenEmbedded Linux (GCC 9.3): `aarch64-oe-linux-gcc9.3`
162+
+ OpenEmbedded Linux (GCC 8.2): `aarch64-oe-linux-gcc8.2`
163+
+ Ubuntu Linux (GCC 9.4): `aarch64-ubuntu-gcc9.4`
164+
+ Ubuntu Linux (GCC 7.5): `aarch64-ubuntu-gcc7.5`
165+
2. Set Target Architecture:
166+
* On your host machine, set the target architecture of your target device by running:
167+
+ `export QNN_TARGET_ARCH="your-target-architecture-from-above"`
168+
* For **Android**:
169+
+ `export QNN_TARGET_ARCH="aarch64-android"`
170+
* For **Linux Embedded**:
171+
+ `export QNN_TARGET_ARCH="aarch64-oe-linux-gcc11.2"`
172+
3. Generate Model Library:
173+
* Run the following command on your host machine:
174+
```
175+
python3 "${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-model-lib-generator" \
176+
-c "${QNN_SDK_ROOT}/examples/Models/InceptionV3/model/Inception_v3.cpp" \
177+
-b "${QNN_SDK_ROOT}/examples/Models/InceptionV3/model/Inception_v3.bin" \
178+
-o "${QNN_SDK_ROOT}/examples/Models/InceptionV3/model_libs" \
179+
-t ${QNN_TARGET_ARCH}
180+
```
181+
* `-c`: Path to the `.cpp` QNN model file.
182+
* `-b`: Path to the `.bin` QNN model file (optional but recommended).
183+
* `-o`: Path to the output folder.
184+
* `-t`: Target architecture to build for.
185+
4. Verify Output File:
186+
* Run:
187+
+ `ls ${QNN_SDK_ROOT}/examples/Models/InceptionV3/model_libs/${QNN_TARGET_ARCH}`
188+
* Verify that the output file `libInception_v3.so` is inside. This file will be used on the target device to execute inferences. The output `.so` file will be located in the `model_libs` directory, named according to the target architecture.
189+
190+
### Step 4: Model Build on Linux Host
191+
192+
Users can set custom options and different performance modes to the HTP Backend through the backend config. Follow these steps to create the necessary JSON files:
193+
194+
1. Create Backend Config File (`be.json`):
195+
* For **Android** with the following content:
196+
```json
197+
{
198+
"graphs": [
199+
{
200+
"graph_names": [
201+
"Inception_v3"
202+
],
203+
"vtcm_mb": 8
204+
}
205+
],
206+
"devices": [
207+
{
208+
"htp_arch": "v79"
209+
}
210+
]
211+
}
212+
```
213+
* For **Linux Embedded** with the following content:
214+
```json
215+
{
216+
"graphs": [
217+
{
218+
"graph_names": [
219+
"Inception_v3"
220+
],
221+
"vtcm_mb": 8
222+
}
223+
],
224+
"devices": [
225+
{
226+
"htp_arch": "v73"
227+
}
228+
]
229+
}
230+
```
231+
2. Create Backend Extensions Config File (`config.json`):
232+
* The config file with minimum parameters specified through JSON is given below:
233+
```json
234+
{
235+
"backend_extensions": {
236+
"shared_library_path": "./libQnnHtpNetRunExtensions.so",
237+
"config_file_path": "./be.json"
238+
}
239+
}
240+
```
241+
3. Place JSON Files in the Model Libraries Path:
242+
* Ensure that the `be.json` and `config.json` files are located in the following path:
243+
+ `${QNN_SDK_ROOT}/examples/Models/InceptionV3/model_libs/${QNN_TARGET_ARCH}`
244+
245+
### Step 5: Prepare the Target Device
246+
247+
1. Open a Terminal on the Target Device:
248+
* Run the following commands to create a destination folder and set the necessary permissions:
249+
```
250+
adb shell "mkdir -p /data/local/tmp"
251+
adb shell "ln -s /etc/ /data/local/tmp"
252+
adb shell "chmod -R 777 /data/local/tmp"
253+
adb shell "mkdir -p /data/local/tmp/inception_v3"
254+
```
255+
2. Push Necessary Files:
256+
* Push `libQnnHtp.so` and other necessary executables, input data, and input list from your host machine to `/data/local/tmp/inception_v3` on the target device:
257+
```
258+
adb push \path\to\qairt\2.32.0.250228\lib\${QNN_TARGET_ARCH} /data/local/tmp/inception_v3
259+
adb shell "cd /data/local/tmp/inception_v3; mv ./${QNN_TARGET_ARCH}/* ./; rmdir ${QNN_TARGET_ARCH}"
260+
```
261+
* For **Android**:
262+
+ `adb push \path\to\qairt\2.32.0.250228\lib\hexagon-v79\unsigned /data/local/tmp/inception_v3`
263+
* For **Linux Embedded**:
264+
+ `adb push \path\to\qairt\2.32.0.250228\lib\hexagon-v73\unsigned /data/local/tmp/inception_v3`
265+
266+
* Continue remaining steps on the target device:
267+
```
268+
adb shell "cd /data/local/tmp/inception_v3; mv ./unsigned/* ./; rmdir unsigned"
269+
adb push \path\to\qairt\2.32.0.250228\examples\Models\InceptionV3\model_libs\${QNN_TARGET_ARCH}\ /data/local/tmp/inception_v3
270+
adb shell "cd /data/local/tmp/inception_v3; mv ./${QNN_TARGET_ARCH}/* ./; rmdir ${QNN_TARGET_ARCH}"
271+
```
272+
```
273+
adb push \path\to\qairt\2.32.0.250228\examples\Models\InceptionV3\data\cropped /data/local/tmp/inception_v3
274+
adb push \path\to\qairt\2.32.0.250228\examples\Models\InceptionV3\data\target_raw_list.txt /data/local/tmp/inception_v3
275+
adb push \path\to\qairt\2.32.0.250228\bin\${QNN_TARGET_ARCH}\qnn-net-run /data/local/tmp/inception_v3
276+
adb shell "chmod 777 /data/local/tmp/inception_v3/qnn-net-run"
277+
```
278+
3. Run the Model & Pull Output Directory to Host Machine:
279+
* On the target device, run the following command to execute the model:
280+
```
281+
adb shell "LD_LIBRARY_PATH=/data/local/tmp/inception_v3 DSP_LIBRARY_PATH=/data/local/tmp/inception_v3; cd /data/local/tmp/inception_v3; ./qnn-net-run --model ./libInception_v3.so --input_list ./target_raw_list.txt --backend ./libQnnHtp.so --output_dir ./output --config_file ./config.json"
282+
```
283+
* Run the following command to pull the output directory from the target device to your host machine:
284+
+ `adb pull /data/local/tmp/inception_v3/output \path\to\qairt\2.32.0.250228\examples\Models\InceptionV3`
285+
286+
### Step 6: View Classification Result
287+
288+
1. Navigate to the Example Directory on Host Machine:
289+
* Run:
290+
+ `cd ${QNN_SDK_ROOT}/examples/Models/InceptionV3`
291+
2. Run the Script to View Classification Results:
292+
* Execute the following command to view the classification results:
293+
```
294+
python3 "./scripts/show_inceptionv3_classifications.py" \
295+
-i "./data/cropped/raw_list.txt" \
296+
-o "./output" \
297+
-l "./data/imagenet_slim_labels.txt"
298+
```
299+
* Ensure that the classification results in the output match the following:
300+
```
301+
${QNN_SDK_ROOT}/examples/Models/InceptionV3/data/cropped/notice_sign.raw 0.152344 459 brass
302+
${QNN_SDK_ROOT}/examples/Models/InceptionV3/data/cropped/chairs.raw 0.281250 832 studio couch
303+
${QNN_SDK_ROOT}/examples/Models/InceptionV3/data/cropped/trash_bin.raw 0.800781 413 ashcan
304+
${QNN_SDK_ROOT}/examples/Models/InceptionV3/data/cropped/plastic_cup.raw 0.988281 648 measuring cup
305+
```
306+
307+
### References
308+
309+
* [Linux Setup](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/linux_setup.html)
310+
* [CNN to QNN for Linux Host](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/qnn_tutorial_linux_host.html)
311+
* [CNN to QNN for Linux Host on Linux Target](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/qnn_tutorial_linux_host_linux_target.html)

0 commit comments

Comments
 (0)