Skip to content

Commit 59f7e1e

Browse files
authored
Whisper App README (#57)
* Added README for Whisper App * Add video to README.md
1 parent c540332 commit 59f7e1e

File tree

2 files changed

+26
-1
lines changed

2 files changed

+26
-1
lines changed

whisper/android/WhisperApp/README.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Whisper Demo App
2+
3+
This app runs the Whisper model with the Qualcomm backend in ExecuTorch. The model itself runs on the Qualcomm HTP (Hexagon Tensor Processor), an AI accelerator in Qualcomm SoCs. You need an Android smartphone with a [supported Qualcomm chipset](https://github.com/pytorch/executorch/tree/main/backends/qualcomm).
4+
5+
### Build the ExecuTorch Android library
6+
7+
Checkout [Whisper JNI Bindings](https://github.com/pytorch/executorch/pull/13525) (if it is not merged already). Build the [ExecuTorch Android library with QNN backend](https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/LlamaDemo/docs/delegates/qualcomm_README.md).
8+
9+
### Export the audio processing and model .pte files
10+
11+
There are two steps, audio processing and the Whisper model (encoder+decoder), which are both done via ExecuTorch.
12+
13+
1) Run the script `extension/audio/mel_spectrogram.py` to export `whisper_preprocess.pte`
14+
2) Run the scrint `examples/qualcomm/oss_scripts/whisper/whisper.py` to export `whisper_qnn_16a8w.pte`
15+
16+
Move these two .pte files along with `tokenizer.json` to `/data/local/tmp/whisper` on device.
17+
18+
### Run app
19+
20+
Open WhisperApp in Android studio. Copy the Android library `executorch.aar` which should have audio JNI bindings and the Qualcomm HTP libraries, into `app/libs`.
21+
22+
23+
24+
https://github.com/user-attachments/assets/ff8c71c5-b734-4ed4-8382-70a429830665
25+

whisper/android/WhisperApp/app/src/main/java/com/example/whisperapp/MainActivity.kt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ class MainActivity : ComponentActivity(), WhisperCallback {
148148
val byteArray = arrayOf(byteBuffer.array())
149149

150150
val whisperModule = WhisperModule("/data/local/tmp/whisper/whisper_qnn_16a8w.pte",
151-
"/data/local/tmp/rohansjoshi/executorch/whisper/tokenizer.json")
151+
"/data/local/tmp/whisper/tokenizer.json")
152152

153153
Log.v(TAG, "Starting transcribe")
154154
whisperModule.transcribe(128, byteArray, this@MainActivity) // this runs runner.transcribe()

0 commit comments

Comments
 (0)