Run Whisper using onnxruntime in C++ with sherpa-onnx

[csukuangfj]

FYI: We have managed to run Whisper using onnxruntime in C++ with sherpa-onnx, which is a sub-project of Next-gen Kaldi.

It supports Linux, macOS, Windows, Raspberry Pi, Android, iOS, etc.

It also provides various bindings for other languages, e.g., C API, Python API, Golang API, C# API, Swift API, Kotlin API, etc.

You can try it using the following colab notebook
https://github.com/k2-fsa/colab/blob/master/sherpa-onnx/sherpa_onnx_whisper_models.ipynb

or try it from within your browser by visiting
https://huggingface.co/spaces/k2-fsa/automatic-speech-recognition

---

Documentation is available at
https://k2-fsa.github.io/sherpa/onnx/pretrained_models/whisper/index.html

[Bensake]

Thanks for your great work. I can't find much information how to run Whisper on Android using your jniLibs. Do you have an example somewhere?

[csukuangfj]

Yes! Please see our android folder. just search for whisper in it.

you can also find pre-built apk for whisper in our doc

[csukuangfj]

To be more specific, please see

VAD+ ASR

1. https://github.com/k2-fsa/sherpa-onnx/tree/master/android/SherpaOnnxVadAsr

It shows how to use silero_vad + whisper for ASR.

You can download a pre-built APK for it at
https://k2-fsa.github.io/sherpa/onnx/vad/apk-asr.html

E.g., sherpa-onnx-1.10.41-arm64-v8a-vad_asr-en-whisper_tiny.apk

Two-pass ASR

2. https://github.com/k2-fsa/sherpa-onnx/tree/master/android/SherpaOnnx2Pass

It shows how to use whisper in a 2-pass ASR, where the first pass uses a small, less accurate but faster, streaming ASR model and the 2nd pass uses a whisper model.

You can download pre-built APK for it at
https://k2-fsa.github.io/sherpa/onnx/android/apk-2pass.html

E.g.
sherpa-onnx-1.10.41-arm64-v8a-asr_2pass-en-small_zipformer_whisper_tiny.apk

Spoken language identification

3. https://github.com/k2-fsa/sherpa-onnx/tree/master/android/SherpaOnnxSpokenLanguageIdentification

It shows how you use a whisper model for spoken language identification.

You can download pre-built APK for it at
https://k2-fsa.github.io/sherpa/onnx/spoken-language-identification/apk.html

E.g.
sherpa-onnx-1.10.41-arm64-v8a-slid-whisper_tiny.apk

---

please see
https://k2-fsa.github.io/sherpa/onnx/android/index.html
for how to use sherpa-onnx's Android exmaples.

[Bensake]

Thank you, I found your examples. Do you know, is it possible to run small LLMs using your jniLibs? For example, huggingface SmolLm models? https://huggingface.co/HuggingFaceTB/SmolLM-135M-Instruct/tree/main/onnx
Because there is an onnx model file. Are the Android binaries compatible with any onnx models or only the ones shown in your examples? 🙏