I put up Whisper with a frontend on a server to use for free

[mayeaux]

I built myself a nice frontend for Whisper and since I'm not using near the full GPU usage I am putting it up online to use for free: https://freesubtitles.ai

It's also open source with the code available here: https://github.com/mayeaux/generate-subtitles

Hope you find it useful and thanks to Whisper for this awesome module!

Hacker news post:
https://news.ycombinator.com/item?id=33663486#33664460

[Abdurrafey-Siddiqui]

Hi @mayeaux
Was wondering if you can incorporate options to tweak the transcription into this amazing work of yours.
Namely initial_prompt and temperature

[mayeaux]

What do they do? Thanks glad you liked the project, check it out just shipped a new feature ;)

[shn-dev-1]

I feel like this is so fast! When I run whisper ra2.aac --model medium the transcription is incredibly slow, but on your site it takes a few seconds. Any idea why this would be?

[mayeaux]

@ssepac probably that it's using a nice GPU server and using the GPUs for the conversion

[glowinthedark]

@ssepac if you happen to run on a mac then this will speed it up a bit: https://developer.apple.com/metal/tensorflow-plugin/

[SomeCodingUser]

Nice work, how are you hosting it?

[mayeaux]

Just paying out of pocket for a server on Vast atm

[tapzzee123]

thank you very much. I was trying to run it for weeks. But I have no programming experience. Thank you so much.

[FurkanGozukara]

thank you very much. I was trying to run it for weeks. But I have no programming experience. Thank you so much.

I have a video that explains how to use whisper it may help. i install on windows from scratch and run it. no programming experience needed

https://www.youtube.com/watch?v=msj3wuYf3d8

[whitesided]

If you're on a Mac you might look into using brew to install pipx. Streamlines installing projects like this where you only want to be able to run a command line tool. Once pipx is set up you just provide it the github url and bob's your uncle.

[rosmur]

@whitesided very interesting! Thanks!!

[vbalashi]

Can you tell me why does it take 20 minutes to complete this via CLI for a 2-minute MP3 file on my MacPro 2020 with 16GB memory:
whisper p1l21.mp3 --language Dutch --model large

while freesubtitles.ai finishes the same thing in just 30 seconds?

What is under the freesubtitles.ai's hood?

[AlbertoGP]

When running on a M1 Mac, you’ll get faster results with this C++ version: https://github.com/ggerganov/whisper.cpp

[glowinthedark]

this also helps: https://developer.apple.com/metal/tensorflow-plugin/

[mayeaux]

When running on a M1 Mac, you’ll get faster results with this C++ version: https://github.com/ggerganov/whisper.cpp

I'll probably change this app to use whisper.cpp, I found that it performs better with a $200/month standard server with 8TB of storage than a $250/month server from Vast.ai with 50 GB of data, have been pretty impressed with it so far

[mflage]

The difference is that I am using a GPU that is designed for machine learning work whereas I'm not sure Whisper is even able to use the GPU of Macbooks, at least for my Macbook Pro M1 it's not able to yet because of some issues with CUDA.

Which GPU is that?

[mayeaux]

The difference is that I am using a GPU that is designed for machine learning work whereas I'm not sure Whisper is even able to use the GPU of Macbooks, at least for my Macbook Pro M1 it's not able to yet because of some issues with CUDA.

Which GPU is that?

Just using A40 servers

[PawBroon]

What stumps me is that you can still, somehow, manage to translate to something else than English.
I haven't been able to do that since a few commits, as if tricking Whisper with an English audio but a --language fr/de/es/it/* and a --transcribe doesn't work anymore.
And I understand it was NEVER meant to be used like that and might have been a fluke all along.
But that was an AWESOME feature and it's weird to so it discarded.

You're using the latest Whisper branch, or did you build upon the one where translation in something else than English was still possible?
Thanks a LOT for that hosting, BTW.

[screenfluent]

Well I am a designer stepping into development area recently so I have developer account just for getting API token. :) You have to try it yourself regarding limits. However if you want the reader to feel like it was translated by human then DeepL is my first choice. Usually without any corrections it is perfect (english to polish and reverse). You look for bulk automatic translation I guess but what I like with manual translation is to be able to click for selected word and choose synonyms or other variations to create simply perfect content. BTW maybe you look for some help with layout for FreeSubtitles?

[mayeaux]

Well I checked and unfortunately DeepL is limited with their languages as well I think they have even fewer than LibreTranslate. I'll stick with LibreTranslate for now but maybe will add DeepL as an optional feature at some point in the future.

Help with the layout how so? You mean design stuff? It's not really on my to-do list at the moment, the UI is working fine for me I don't have any plans to improve it at the moment lol, do you have some suggestions?

[glowinthedark]

paid deepl offers unlimited translation only in the web flavour, the free API access gives you 500,000 chars per month for free; the paid API plan includes a fixed number of chars per months and if you exceed that you have to pay for the extra chars.

From my tests with freely available neural models for machine translations I'd say facebook's fairseq is very decent but has few languages. There is also a huggingface transformers port of the m2m100 model which I found to be pretty good with support for tons of languages, probably on par or better than google translate but not as good as deepl. HelsinkiNLP models are also ok, but quality would depend on the language pair. If you want an easy python wrapper for those models have a look at easynmt. All these are very resource hungry.

[mayeaux]

paid deepl offers unlimited translation only in the web flavour, the free API access gives you 500,000 chars per month for free; the paid API plan includes a fixed number of chars per months and if you exceed that you have to pay for the extra chars.

From my tests with freely available neural models for machine translations I'd say facebook's fairseq is very decent but has few languages. There is also a huggingface transformers port of the m2m100 model which I found to be pretty good with support for tons of languages, probably on par or better than google translate but not as good as deepl. HelsinkiNLP models are also ok, but quality would depend on the language pair. If you want an easy python wrapper for those models have a look at easynmt. All these are very resource hungry.

Thanks for the info I will be checking these out very shortly

[mayeaux]

EasyNMT worked really well for me with the m2m_100_1.2B model, thanks for informing me of these I will be deploying these to prod soon.

[an303042]

Thank you for this!
I've been pulling hairs trying to get whisper to run on my gpu locally in order to transcribe some file. Just ended up using your site.
I feel a little like a cheater but at least I got the job done. :)

[mayeaux]

Thank you for this! I've been pulling hairs trying to get whisper to run on my gpu locally in order to transcribe some file. Just ended up using your site. I feel a little like a cheater but at least I got the job done. :)

Hey whatever works! Lol, glad it was useful for you

[TeKnoID030]

Hi This is great, I just tried it out. And I thought it didn't work because I didn't see any prompt or status update. It is a bigger file, so it is taking some time. But maybe good to have some status update so people now it is still working ;-)
Awesome project! Thanks

[TeKnoID030]

Actually it seems stuck now... it has been on this for a while now:
Processing..
finishedProcessing

Latest processing data (process 2)

[100%] 00:00 Remaining, Speed 200.50f/s

[mayeaux]

@TeKnoID030 sorry there was an issue with the Vast.ai server, something happened with their docker instances where the GPUs weren't being detected so it was running off the CPU, should work now if you try again. Vast is a bit unstable but by far the cheapest out there

[glowinthedark]

@mayeaux: you could probably significantly cut down GPU costs by running the cpp optimized version of whisper on CPU only. That version has no GPU support but runs a lot faster than the original whisper (especially on multicore boxes); the author also provides demo code to run it on an iphone in real time.

[mayeaux]

Yup, actually I am going to try this out today, fingers crossed that it's performant, using these GPU Cloud Services has been a nightmare would be great if I could use just a 'normal' dedicated server

[benjieperez]

May i know how you calculate the percentage of being transcribed?

[mayeaux]

May i know how you calculate the percentage of being transcribed?

It comes from using Whisper with verbose=False, it's calculated by the Python library and put to stdout

[lxsssssss]

Can you tell me how to read the transcription progress and show it, thank you very much

[mayeaux]

Can you tell me how to read the transcription progress and show it, thank you very much

Use spawn with Node.js and process the data as it comes through stderr is how I do it

[IlianP]

It would be a cool to see the number of queued files before uploading a file. That way you can decide if it's worth the wait.

[mayeaux]

It would be a cool to see the number of queued files before uploading a file. That way you can decide if it's worth the wait.

It'll come, but I increased the number of processes from 1 to 5 so the wait time should be better now

[twitchk]

Wow, nice work thanks. I tried your web interface and i was wondering how i could download the subtitles into a .txt file. That option wasnt given. I'd be grateful if you could add this.

[mayeaux]

Wow, nice work thanks. I tried your web interface and i was wondering how i could download the subtitles into a .txt file. That option wasnt given. I'd be grateful if you could add this.

When you're on the player page there is a button to download the .txt version

[robinkoehler]

This is so cool! Thank you for making that available. I've tested it against my local install on a 3090, and the speed is still pretty great.

The thing that your implementation does far better than vanilla whisper is how you are formatting the output text, even just in the .txt version. Locally, I can just get ugly line breaks where there shouldn't be any, and your output is so nicely segmented.

Is that some magic you've added manually, are you doing post-processing, or are you using some fancy whisper argument that I am not aware of?

Thank you again for providing this service and I appreciate your time and response!

Best wishes,

Robin

[mayeaux]

Nothing magic or post processing (except for a couple languages to use easier scripts). This does just use 'vanilla Whisper' it's just a wrapper around the command line using Node.js. Glad you've found it useful!

[hgfriver]

Nice Work!!!!! thanks!!!

[z00g00l]

Hey everyone! Has anyone tried this for song transcription - I've uploaded a few mp3 songs but they haven't processed.
P.S I haven't run the model locally yet was just trying it out on this great UI beforehand!

Thanks for making this available again!

[meaz]

Thanks a lot, for how long are uploaded files kept on the server?

[mayeaux]

The media file is stored for 1h the rest are available permanently or at least until I change the server which happens from time to time. With paid use you can store the media file permanently as well

[marcvie]

To run OpenAI Whisper LARGE model, how does the Nvidia RTX 4090 compare to this setup on AWS - NVIDIA A10G Tensor Core GPU, g5.xlarge with 16GB RAM? Can I expect faster or slower transcription with the 4090?

[mayeaux]

I've never used AWS so I can't compare. 4090s work really well though, probably 4 minutes of transcription per 1 minute of GPU use using the large model

[marcvie]

4:1 and sometimes 5:1 (for slow dictation) is about the turnaround time I get on my AWS setup. Is there any way I can solve the Whisper hallucination issue which especially occurs in the last sentence of transcripts?

[marcvie]

If cost were no issue, which would be the best GPU to buy to run the Whisper LARGE model locally the fastest?

[mayeaux]

If cost were no issue, which would be the best GPU to buy to run the Whisper LARGE model locally the fastest?

I can't say for sure but I think faster-whisper is a good option to speed up the transcription process by a large margin (at least 4x in my experience), that's what I'm using now

[guillaumekln]

Hey, nice work!

You might be interested in this project to reduce cost: https://github.com/guillaumekln/faster-whisper

I also saw that you consider integrating the NLLB-200 model (mayeaux/generate-subtitles#14). The CTranslate2 library that is used by faster-whisper, can also run the NLLB-200 efficiently. See an example here and here.

[mayeaux]

Hey, nice work!

You might be interested in this project to reduce cost: https://github.com/guillaumekln/faster-whisper

I also saw that you consider integrating the NLLB-200 model (mayeaux/generate-subtitles#14). The CTranslate2 library that is used by faster-whisper, can also run the NLLB-200 efficiently. See an example here and here.

I'm now using faster-whisper for freesubtitles.ai and it's a miracle how well it works! I was worried that it would be cost prohibitive to try and scale up the project using Whisper's stock inference but faster-whisper runs so quickly that I am pretty confident I could scale this up easily now to meet a big demand, it's really amazing thanks a million for implementing it and putting it out ! 🙌

[kanazei]

You are the best!
Could you add a large-v1? For Russian it works better.
And initial_prompt, it also improves transcription.

[mayeaux]

You are the best! Could you add a large-v1? For Russian it works better. And initial_prompt, it also improves transcription.

I've never used initial_prompt before so it's not a super high priority for me, how do you use it normally?

And I want the app to be as non-technical as possible so I'm not sure if I will add v1 and v2 as a selection, also not really a priority for me atm but Ill consider it.

[Karl1b]

I just want to say thank you.
I started whisper on my Laptop and a medium model and it takes ages. So I had the chance to use the large model. ;-)

[mayeaux]

I just want to say thank you. I started whisper on my Laptop and a medium model and it takes ages. So I had the chance to use the large model. ;-)

It uses faster-whisper now so it should run really quick now 😁

[IlianP]

First of all, it's amazing and works really fast with faster-whisper (even with the large model) so: thanks a lot! 🚀
The player works like a charm as well, especially the feature to display both languages at the same time. 👍
But when I tried the translation on a 30min mp3 with German audio, I found an issue with the text of the translation. Can you help me find out whether it's related to freesubtitles.ai or the Whisper model in general? So, the problem in the translated text (German to English) seems to be that only every single text line is translated one by one. But in long sentences that are cut in two, it should (obviously) translate the whole sentence instead of only parts of it. [For those interested: Especially in the German language, where the verb tends to be all over the place within the sentence, it sometimes looses the context completely.] So how is the translation done technically? Are you using Whisper's own translate feature?

[mayeaux]

First of all, it's amazing and works really fast with faster-whisper (even with the large model) so: thanks a lot! 🚀 The player works like a charm as well, especially the feature to display both languages at the same time. 👍 But when I tried the translation on a 30min mp3 with German audio, I found an issue with the text of the translation. Can you help me find out whether it's related to freesubtitles.ai or the Whisper model in general? So, the problem in the translated text (German to English) seems to be that only every single text line is translated one by one. But in long sentences that are cut in two, it should (obviously) translate the whole sentence instead of only parts of it. [For those interested: Especially in the German language, where the verb tends to be all over the place within the sentence, it sometimes looses the context completely.] So how is the translation done technically? Are you using Whisper's own translate feature?

It's using Google Translate, but yes it translates them with newlines in between based on the vtt input. You're right that it could be optimized to have better translations by merging the vtt lines that can be reasonably inferred as being a complete sentence (aka no comma, question mark or period) but I hadn't really thought about that solution until you mentioned this. I'll put it on the to-do list, and glad you liked the platform! Better timestamps and subtitles based on the word_timestamps parameter will be deployed soon, probably at some point today.

[mayeaux]

First of all, it's amazing and works really fast with faster-whisper (even with the large model) so: thanks a lot! 🚀 The player works like a charm as well, especially the feature to display both languages at the same time. 👍 But when I tried the translation on a 30min mp3 with German audio, I found an issue with the text of the translation. Can you help me find out whether it's related to freesubtitles.ai or the Whisper model in general? So, the problem in the translated text (German to English) seems to be that only every single text line is translated one by one. But in long sentences that are cut in two, it should (obviously) translate the whole sentence instead of only parts of it. [For those interested: Especially in the German language, where the verb tends to be all over the place within the sentence, it sometimes looses the context completely.] So how is the translation done technically? Are you using Whisper's own translate feature?

I updated the translation functionality with what I think is a pretty clever algo, check it out and let me know how it runs now

[IlianP]

I had started testing the new translation after you put it up (and the first impression was a great improvement!) but it seems to me it's been reverted back to the old "line by line" translation now? At least that's what I got when I tested it again just now.

By the way, is there no model selection anymore? What's the standard model that is used now?

[mayeaux]

I had started testing the new translation after you put it up (and the first impression was a great improvement!) but it seems to me it's been reverted back to the old "line by line" translation now? At least that's what I got when I tested it again just now.

By the way, is there no model selection anymore? What's the standard model that is used now?

Try again, it should be working again. The only issue is that occassionally Whisper will not give punctuation during the transcription, in this case I can't compile the segments into full sentences before the translation, but try it again it should be working well now I did some optimizations with the VAD. Also have support for DeepL and Yandex translations as well now!

[coolcoder613eb]

this take 2 mins to transcribe a file that wasn't finished after running it overnight on my 2020 m1 MacBook Air

[jeni1616]

Hi @mayeaux

You said $200 or $250 for hosting on vast ai, but If i check price of A4 there: https://cloud.vast.ai/create?utm_medium=cpc&utm_target=&device=c&gad=1&gclid=CjwKCAjwjaWoBhAmEiwAXz8DBaD09zw8AGde37fA5Y8xC-_oMWhk8IYXEpg3ZG6rV_Qndzkyg9-_HRoCPi8QAvD_BwE&utm_content=&placement=&utm_group=&adposition=&utm_source=google&utm_campaign=20556932576_

it's much more than what you said? even if i take lowest A40, it's 0.403/hr so 0.403 * 60 * 24 = 580$ just for 1 day. can you explain bit more? struggling to host my other generative model (not related with Whisper)

Thanks!

[PaulRosset]

Why are you multiplying by 60? It should be 0.403/hr * 24 = 9.672 and * 30 for a month = 290.16

[jeni1616]

@PaulRosset Lol, Thanks, I was in the assumption that it was per minute price ;p

[hnorgaar]

Nice idea and option to have for subs, but now just a scam to steal our money and maybe get a lot of movies uploaded to his server for free

[rohitkrishna094]

I believe data is deleted after 30 mins. At least thats what it says.