Livestream transcription/translation

[fortypercnt]

I implemented a CLI tool that can use whisper directly on livestreams in ffmpeg, transcribing every couple of seconds. You can find it here. By default it uses streamlink to automatically get .m3u8 files for various streaming sites, but URLs can also be passed to ffmpeg directly.

[Mijawel]

I was wondering if your tool takes into account previous texts so that the chunks of transcribed text aren't disjointed?

[fortypercnt]

Unfortunately whisper does not provide a convenient way of dealing with real time audio. It does give the option to condition the model on previous text by giving a prompt, but in my limited testing the results were better without that. It often repeated itself and got stuck in loops. Mileage may vary when using windows longer than just a few seconds.
I might try to build something more sophisticated (e.g. based on sliding windows as described in #117 (comment)) when I have more time.

[jianfch]

verbose=True already prints the segments as they are being transcribed\translated.
so wouldn't it be more simple to just use verbose=True, instead of chopping the audio up even more and printing the results in blocking manner which makes results disjointed.
if you want direct access to the data as it is being decoded, you can just add a hook to add_segment()

[fortypercnt]

The problem is with real time audio, there is only one segment for each call to transcribe/decode, which contains the last few seconds of audio captured from the live stream. Or maybe I am missing something.

[jianfch]

there is only one segment for each call to transcribe/decode

oh i see, so wouldn't make a difference.
you can try during inference to fill in the padded portion with proceeding audio packets as it coming in. then shifting the window upon getting end-of-transcript token or reach 30 sec of unpadded audio. this way, you can mimic feeding a continuous stream of audio to avoid disjointed results.

[nyadla-sys]

Please see my project below, which uses the Whisper Tiny Tflite Model to implement audio streaming..
Streaming using TFLite model

[nyadla-sys]

Refer the below github repo for Android example, which uses whisper-tiny.en.tflite model
https://github.com/nyadla-sys/whisper.tflite