Roadmap to perform speaker diarization

[anon747]

Firstly, thanks a lot for providing such a concise tool to play around with audio! Big ups! :)

I was exploring the pyannote project and got confused with so many repositories that provide different functionalities. I wanted to know if my understanding of the way to use this package is correct.

Let's consider the following scenarios:-

1. If I have an audio clip (say from a college debate competition) involving 3 speakers.
   Does it make sense to use pretrained models from pyannote.audio.hub to identify who speaks when?
   Or should I retrain/fine-tune all of the pipelines (speech activity detection, speaker change detection, overlapped speech detection, speaker embedding) on audio data in which these 3 people speak?
   I ask this because when I used the pretrained diarization pipeline from pyannote.audio.hub, I received random speaker labels like A, 101, 99.

2. Does it make sense to train pipelines on data from specific domains and expect them to perform well on other data from the same domain?
   For example, if I train the above-mentioned pipelines on rap music by Eminem, Lil Wayne, and Kanye West, am I correct to use these pipelines to try and perform speaker diarization on a rap song in which Pitbull and J Cole rap?
   Or will the trained pipelines be applicable to only Eminem, Li Wayne, and Kanye West's rap?

Please excuse me if these questions sound silly/stupid. I just want to understand the best way to use this fantastic tool that you have built.

[hbredin]

1. If I have an audio clip (say from a college debate competition) involving 3 speakers.
   Does it make sense to use pretrained models from pyannote.audio.hub to identify who speaks when?

First thing first. Are you looking for speaker diarization or speaker tracking/identification?

pyannote.audio does speaker diarization out of the box and can be used for speaker identification with more (coding) efforts on your side.

   Or should I retrain/fine-tune all of the pipelines (speech activity detection, speaker change detection, overlapped speech detection, speaker embedding) on audio data in which these 3 people speak?

Your question makes me think that you are actually interested in speaker tracking and not speaker diarization.

   I ask this because when I used the pretrained diarization pipeline from pyannote.audio.hub, I received random speaker labels like A, 101, 99.

That is expected as actual labels do not really matter in speaker diarization (see definition above).

2. Does it make sense to train pipelines on data from specific domains and expect them to perform well on other data from the same domain?
   For example, if I train the above-mentioned pipelines on rap music by Eminem, Lil Wayne, and Kanye West, am I correct to use these pipelines to try and perform speaker diarization on a rap song in which Pitbull and J Cole rap?
   Or will the trained pipelines be applicable to only Eminem, Li Wayne, and Kanye West's rap?

Yes, it makes sense to fine tune models/pipelines on a specific domain and is even recommended.
The larger and more diverse your training set is, the better it will generalize (to other speakers in particular...)

[anon747]

Hi @hbredin!

Thank you for the timely response!

I now understand that pyannote-audio is capable of speaker diarization right out of the box. I am interested in speaker diarization and not in speaker tracking - I just wanted to understand the capabilities of the library that you've created. I am more interested in examples like the one I described in scenario 2 (training for speaker diarization on the rap music of artists A, B, C and using those models to diarize rappers in songs of other artists).

Could you please tell me if my approach is correct in case I want to perform something as described in scenario 2?
To make sure that the models adapt to a particular domain (for example, rap music) I will need to fine-tune the existing pre-trained models or retrain them entirely. I will have to take the following steps:-

1. Re-train/fine-tune the following models (as described here) -
   a. speech activity detection
   b. speaker change detection
   c. overlapped speech detection
   d. speaker embedding

2. Finally, I would re-train/fine-tune the speaker diarization pipeline while using the models re-trained/fine-tuned above, as described here.

Thanks once again for your help! :)

[hbredin]

I'd say the most critical parts are 1a, 1b and 2.
You do not need to do 1c because overlapped speech detection is not used in the diarization pipeline.
I would not recommend doing 1d because you'd need a large number (100+) of rappers to really benefit from it.

Also, if you have a large dataset of annotated rap music that you can share, I'd be curious to play around with it.

[anon747]

Thanks a lot for your inputs!

We (me and other members of my team) will be starting with the annotation part of our project from the next week. I will only include 1d into the picture in case we are able to compile a large enough dataset (100+ speakers) like you'd mentioned.

Also, I was using the rap music domain as an example so unfortunately, our domain will not be as cool. However, we do aim to annotate audio clips from a few different domains. Since our work will be public, I will definitely share all annotated datasets with you, after we complete the project. Hopefully will get a chance to work on rap music too, in the future. :D

Thanks again!

[Yagna24]

Hi @anon747, I am working on the same pipeline but for a different use case. Can you share pieces of your work so I could understand in more depth about the tool...

[Yagna24]

I'd say the most critical parts are 1a, 1b and 2. You do not need to do 1c because overlapped speech detection is not used in the diarization pipeline. I would not recommend doing 1d because you'd need a large number (100+) of rappers to really benefit from it.

Also, if you have a large dataset of annotated rap music that you can share, I'd be curious to play around with it.

Hi @hbredin can you share the link to main diarization pipeline that has the audio separations with their time frames or anything close to that ? I need to get the audio separated and stored for each speaker.

[EddyGiusepe]

Hello everyone!
I need help with the following task, please, I'm currently researching, Speaker Identification. For that I'm trying to implement a Neural Network in which I will receive the embeddings of my audios (Dataset_SI). For this I want to take advantage of the Speaker Embeddings layer, this layer has already been trained with Dataset_SI for the Diarization task. My question is: should I retrain this layer for a Speaker Identification task? knowing that the new audios involve speakers from this Dataset_SI. If that were the case, I would have to do the labeling, pass the files: .rttm, .lst and .uem, right?
... I'm thinking here: I would have to use the Diarization Pipeline, too, right? to get the audio segments of each speaker, right? and save these embeddings (obviously), right?

Thank you very much!!!

[Yagna24]

Hi @EddyGiusepe your project sounds really interesting. Being fascinated by the same I want to ask how to implement the embedding layers. If you could share any resources.

[EddyGiusepe]

Hello @Yagna24!!!
Sorry for the delay in responding.
I'm still studying that part, but I'm already understanding the mechanics better. I am taking advantage of the documentation of a work colleague and the Pyannote documentation above all. My colleague's approach was for diarization. So, in principle, have the folders well organized. A main folder (obviously), Dataset, configuration files (config.yml), model folders (obviously with their config.yml), database.yml (this protocol is very important; for your Data), etc. This is important because the training, validation and testing will be via command in the GoogleColab.

That is what I'm understanding until now.

In the next message I'm going to send you more or less the structure that I found here in Pyannote (obviously complemented with the document that I read from my colleague, too) and some links that are helping me to understand more and more the flow.

If you have any idea (answer) regarding my question, you would help me a lot. I need ideas :).

Thanks!

[EddyGiusepe]

The order of my folders are (for example):

• Pyannote_main_1
  • MUSAN_Dataset
  • AMI_Dataset
  • Your_Dataset
  • Pyannote_main_2
    • pipelines
      • speaker_diarization
        - config.yml
    • models
      • speaker_embeddings
        - config.yml
      • speech_activity_detection
        - config.yml
• Pyannote_code.ipynb
• database.yml

You should understand how to configure (I'm assuming you don't already know) .yml files. Both for the layers that you're thinking of training and for the use of the Dataset via protocols. Then I send you the links that I mentioned above and from which I'm learning.

Links of studies:

• Changing Sampling Rate Fails in SAD #300 (comment) (For example: for a speechActivityDetection task)

• DomainAware has an error on develop branch #242 (comment)
• ImportError: cannot import name 'Config' from 'omegaconf #925 (comment)
• Embeddings with short time windows #458 (comment)
• Possible bug in TripletLoss- Sampling:hard  #112 (comment)

and for the database.yml file you can study the following link:

• https://github.com/pyannote/pyannote-database/blob/develop/tests/data/database.yml

Until then @Yagna24 .