Whisper Implemented as a Batch Endpoint on Azureml Studio

[hashefa]

Hi,
I have a very large audio dataset (~3000 hours) that we want to convert to text using the wonderful model (which gives us much better results than the azure STT service)
I want to process the data on the Azureml studio, in the fastest processing time, and with minimal cost.
I'm looking for someone who has done something similar or anyone with advice for such a task.
Help will be much appreciated.

[richardburleigh]

How I did a similar project:

1. Uploaded the audio to a storage bucket
2. Created a simple API to track the file URLS, progress, and results
3. Wrote a Python script to bring the above components together and run Whisper
4. Build this into a Docker container to deploy on vast.ai very cheaply

I now have about 10 GPUs running to process the audio quickly.

[richardburleigh]

If you really need help, you can contact me by email (on my profile).

[hashefa]

Thank you.
I wrote you an email.

[richardburleigh]

@hashefa check your spam folder :-)

I'll post some of my reply here in case it helps other people:

The first point is that the original Whisper code performs really badly on CPU. However, there is a nice project to make it work well on Apple Silicon if you have a modern mac: whisper.cpp

Otherwise, the main cost factor is GPU time. I use vast.ai because it's very cheap to rent high powered GPUs, so it could be a nice option. All of the major cloud providers charge way too much for their GPUs, so it would just be wasting money.

An extra cheap option for GPU access is to use Google Colab. You can even use the free version, but it may shut down if you are not active in the notebook. You would need to subscribe to the Pro tier for background execution, although it's pretty cheap anyway.

Regarding your project, there are many ways to achieve your desired outcome. I was looking at the Azure API and it seems it's possible to get a list of the blobs in storage.

You could use this feature in the following way:

1. Get a list of all mp3 files in the blob storage
2. For each file, check if a txt file with the same name exists already
3. Download the mp3 if a txt file doesn't exist
4. Run whisper and upload the txt file to your blob storage
5. Repeat

You mentioned splitting the files by silence, however I don't recommend this. Whisper is really great because it 'understands' the context of the speech, but this will be lost if you split the audio, so it's best to do each file completely, and on a GPU you would only process one file at a time because of the VRAM requirements.

In Python, you should be able to do this in a single script / notebook.

If you don't know Python, you could use the vast.ai option to rent a GPU with enough storage to hold your files, then connect to it by SSH and run a command like this to process all the files:

find . -type f -name "*.mp3" -print0 | xargs -0 -I {} whisper {} --language English --model medium

You would need to connect using a tool like FileZilla for transferring the files.