Best model and settings for (not clean) audiodramas?

[Pocokk]

Hey there!

As my hearing is not so great, the voices usually get lost amidst of various sound effect (gunshot, street noise, background distortions etc) and I don't understand the context of certain audiodramas, its just a big mess for me sometimes.

That's why I'm asking here and now the community's help, which model and settings should I use for those type of transcribing, when the source is not clean, it includes a lot of extra sound effects and overall can be messy to hear out words even for a mother-tongued English speaking person in the first place.

Has anyone figured out the optimal usage in that case?

I appreciate all the help here and thank you so much in advance!

[glangford]

If you haven't had success transcribing noisy audio with the large-v2 model, using a tool such as demucs to separate the spoken portion prior to running whisper might be a good first step.

You can use a command similar to this to separate vocals and 'everything else':
python -m demucs --two-stems vocals --mp3 -o demucs noisy.mp4
(The --two-stems=vocals option allows separating vocals from the rest of the accompaniment (i.e., karaoke mode))

Spleeter is another option (I haven't used that).

demucs is headlined as providing "music source separation", I don't know how well it would perform on your particular audio with FX noise so it's worth experimenting.

See also

• Doesn't seem to be working WITH SONGS? Keeps duplicating transcripts on several timestamps #1833
• How to use "Whisper" to detect whether there is a human voice in an audio segment？ #1907 (comment)

https://github.com/adefossez/demucs

[Pocokk]

Well, the first try was a disaster: I set it up, it worked until 100% progress then suddenly the whole system frozen with maximum CPU/HDD load, here is the photo I took 20 minutes later. As there wasn't anything in the (Windows) Event Viewer logs, I can't provide a source about what went wrong, but now I'm bit reluctant to try again. I've also looked it up and couldn't find a related issue in the past, which is a bit disturbing.

[phineas-pta]

demucs has tendency to use a lot of ram (not vram) which freeze pc with not enough ram

anyway demucs and others like spleeter are for dealing with songs, so sound effect in your case is not suitable, there's no automatic solution for now, only manual editing

[Pocokk]

That's sad to read, thank you.

[glangford]

fyi

• device=cpu, Too high memory usage when processing long audio facebookresearch/demucs#498