ScribeAI - a native ios app for Whisper with real-time transcribing

[RRUK01]

ScribeAI.mp4

I've been inspired by the whisper project and @ggerganov and wanted to do something to make whisper more portable. So I've made ScribeAI a native ios app that runs whisper (base, small & medium) all on-device.

You can get here: https://apps.apple.com/app/scribeai/id6450321299

It includes the following:

• Live mode (as above) as well as post-speech mode, which processes speech when there's been a significant pause to save on battery.
• Base, Small & Medium(!) models all on-device. The demo above is using the Base model and the video is a true live demonstration (no editing involved).
• The models are all Q6_K quantised for the decoder and the encoder runs on CoreML. I experimented a lot with this and found this had the highest accuracy. I've open sourced the code for K_Quants for Whisper (Metal Decoder Inference, K-Quants added, Sync'd with latest GGML ggml-org/whisper.cpp#1074), I've been having a look at the making the decoder run on Metal as well and will open source this too if I can crack it.
• Searchable recordings, you can search through your recordings and it'll pull up all the recordings with your keyword in the transcript.
• Everything is on-device. No data is collected, as such it even works offline and in airplane mode - true portability.
• On-device translation supported (to English).

In the future we're looking to add:

• Speaker diarization taking inspiration from [1] and the https://github.com/akashmjn/tinydiarize library
• More accurate word level timestamps
• More export options
• Fine-tune the models further, currently collecting data to support this and money from any subscriptions will pay towards supporting this effort

Hope you enjoy it and happy to hear any feedback or feature requests at [email protected]!

[1] https://arxiv.org/abs/2305.18747

[chuck-fyn]

Hey RRUK01, this is awesome. Would you be willing to share the source code for this? I'm trying to implement live transcription using the whisper model, but all the examples I am seeing need to record the audio first and then send to to the model. Curious how you managed to route the live audio to the model. Thanks!