Whisper-powered lyrics extraction from songs 🎵

[eshiofune]

I wrote an article here describing how to make use of either the Whisper API or models to extract song lyrics from audio files.
I was able to get up to 89% similarity with the lyrics found on Google for the Adele song Set Fire to the Rain.

The code is available here.

[phineas-pta]

there're already so many tutorials/articles on how to use whisper, to further expand your article and improve accuracy, you should try out source separation models (Spleeter or Demucs) to extract vocals, VAD (SileroVAD or pyannote-audio) to remove silence / background-music-only segments, creating subtitle for karaoke, etc.

[eshiofune]

Thanks for the suggestions !
Creating subtitles for karaoke is right up my alley. Will look into that

[EtienneAb3d]

See:
https://github.com/EtienneAb3d/WhisperHallu

[eshiofune]

See: https://github.com/EtienneAb3d/WhisperHallu

@EtienneAb3d Yes I also noticed the hallucination effect around the end of the song and reading the thread you referenced in your repo and of course the repo itself shows some interesting approaches to reduce this effect. Thanks ! I'll take note of this for improving accuracy.