Whisper Demo App

[rohansjoshi]

Demo app showcasing Whisper running on-device

Both audio processing (STFT+Mel filterbank) and the actual Whisper model are exported to .pte files with ExecuTorch, and run in the app

The app does the following:

1. Press button, record audio
2. Save input as single-channel PCM
3. Convert and save to .wav file (manually write .wav header), with 2 bytes per sample
4. Open .wav file, read byte array two bytes at a time and convert to float array (little endian is convention for .wav)
5. Convert float array to Tensor using ET Tensor binding, pass it through Module which wraps audio processing .pte
6. Since the Qualcomm Whisper Runner reads raw bytes as input (maybe this should be changed), we convert the output to a byte array. Make sure it is in little endian order (this is the runner's convention)
7. Pass array into WhisperModule which wraps the runner which runs the actual Whisper model .pte (encoder+decoder)

To build the app, you need to

1. Export both .pte files using scripts in ExecuTorch. For audio preprocessing run extension/audio/mel_spectrogram. For QNN whisper, run the script examples/qualcomm/oss_scripts/whisper. Move both pte files to /data/local/tmp/whisper on device.
2. Checkout Whisper JNI bindings PR, build the Executorch Android library with Qualcomm backend, and save executorch.aar
3. Copy executorch.aar into app/libs
4. Build the app in Android studio

[kirklandsign]

So we bundle the wav file with the apk?

[rohansjoshi]

In the writeWavFile function the wav file is written, runWhisper runs afterward assuming it already exists