Audio transcription with speaker labelling using speaker samples. Using Whisper & Pyannote #2609
Replies: 10 comments 23 replies
-
|
how much time will it take to process 5 min audio clip? |
Beta Was this translation helpful? Give feedback.
-
|
5 min clip shouldn't take long with a GPU, possibly 2 minutes? Without a GPU probably a bit longer. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @alunkingusw Can you show me your "pip list" or list of libararies? At least cuda/torch things, I am suspecting mine to be too recent and hurting and making my whisper SLOWER. I think. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, my requirements.txt that I use on my docker image is here: Audio processing and transcriptionopenai-whisper I think that is everything! |
Beta Was this translation helpful? Give feedback.
-
|
no I need the versions. Please @alunkingusw (like what torche version? What torchaudio version? etc ectc). What about cuda etc if there is. Thanks. When I do pip list on my side it gives me all libraires with versions |
Beta Was this translation helpful? Give feedback.
-
|
Google Colab has all the libraries pre-installed, I ran pip list on that and it had this huge list. Anything else I install is just the latest version. The only package where I specify a version is numpy, use version 1, so that the numpy.nan in pyannote doesn't crash. Package Version absl-py 1.4.0 |
Beta Was this translation helpful? Give feedback.
-
|
@alunkingusw Scriberr looks promising. rishikanthc is working on v1.0.0. It uses whisper and pyannote as well. It can
There are no subtitle features but if there are skilled devs reading there's no reason it couldn't get added. Give it a look. |
Beta Was this translation helpful? Give feedback.
-
|
Cool! I'm packaging mine into software that will allow transcription of groups. It is designed to help a supervisor keep track of meetings between groups, so will transcribe audio and summarise using LLMs also, with a focus on individual contribution to the group effort. I'll check out Scriberr! |
Beta Was this translation helpful? Give feedback.
-
|
https://drive.google.com/file/d/13WRf4UUCUBQ0NSzdtOfMjy90WTiWve1Z/view?usp=drivesdk |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, you'll need to provide some more context to this comment! |
Beta Was this translation helpful? Give feedback.
-
|
Hello, can I give you a test subject ( a youtube video) so we can see if it really work well? (It's a video with 2 speakers, where they sometimes speak over each other, as in one interrupt the other to say something then the other continues talking) |
Beta Was this translation helpful? Give feedback.
-
|
Sure, send a link and I'll see how it runs. |
Beta Was this translation helpful? Give feedback.
-
|
Do you have an email or something else where I can send the link to the video? Thanks (or even some linkedin or anything) |
Beta Was this translation helpful? Give feedback.
-
|
Follow up questions:
|
Beta Was this translation helpful? Give feedback.
-
|
Hi,
|
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
My linkedin username is alunking so you should be able to find me that way! |
Beta Was this translation helpful? Give feedback.
-
Ok sent you a request (with a note that contains a message) just answer me or add me so I can continue and send you the video link (I dont wish to send it here) then you can unadd me if you wish, I just wish to send my request outside of github. Thanks so much. |
Beta Was this translation helpful? Give feedback.
-
|
Did you receive it? |
Beta Was this translation helpful? Give feedback.
-
|
No I haven't yet sorry |
Beta Was this translation helpful? Give feedback.
-
|
Hey, for some reason my account was blocked :(. You can keep me informed on en mail i guess I will try to share it later |
Beta Was this translation helpful? Give feedback.
-
|
Quick additional question: |
Beta Was this translation helpful? Give feedback.
-
|
I think 10-15 seconds is ideal for an embedding. I can't remember where I read that so feel free to have a look if there is another length recommended. |
Beta Was this translation helpful? Give feedback.
-
|
Ok additional feedback: |
Beta Was this translation helpful? Give feedback.
-
|
So whisperX looks like it already integrates with Pyannote if you want it to. It will label the speakers, but not match them to an embedding to give them their name, which is what I am looking at doing. I haven't looked at their code to try this. If you use WhisperX by itself, it doesn't care about speaker attribution, it just cares about translating whatever is said to text. It is Pyannote that attributes words to speakers, and any problems with speaker crossover is likely down to Pyannote. Whisper has been trained on a much larger dataset and I'm not surprised if Pyannote struggles with speaker crossover. I haven't looked at it enough to know what it is like. |
Beta Was this translation helpful? Give feedback.
-
No problem, just know that it is at least 15X faster and does no hallucination if you use the right model! |
Beta Was this translation helpful? Give feedback.
-
File "/root/miniconda3/envs/whisper/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 547, in from_pretrained |
Beta Was this translation helpful? Give feedback.
-
|
No, I haven't had that error, but I don't use AutoModel to load the model - what is the model name you are passing? |
Beta Was this translation helpful? Give feedback.
-
|
Hey @alunkingusw, this is great! a review UI to fix speaker attribution, We'd love to pilot this with you or any contributors, just a couple sample recordings and we’ll run the pipeline, give you clean exports, and show how reviewers can clean up errors fast. LMK if you'd be open to a short run! |
Beta Was this translation helpful? Give feedback.
-
|
Hi, this sounds great - similar to what I am doing I guess! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I've written some code that I am using in a project and I thought I would share it with you.
The code transcribes an audio file using Whisper, then diarises (sorry for the lack of z, I am British) using Pyannote.
Once the two processes are complete, the result can be output, but my code then takes known speakers from sample clips, generates embeddings and compares them to the speakers identified on the audio.
If a speaker is recognised, then they are labelled in the output. If a speaker is not recognised, they are removed and the final transcription has them labelled as [None]. This removal is optional and you can comment that out in the code if you want. I found it useful to filter out things like musical interludes in a podcast, and focusing on the known speakers.
Here's the code which I had run in Colab
https://gist.github.com/alunkingusw/2eb29682a98f94a714d10080ed0f4896
Beta Was this translation helpful? Give feedback.
All reactions