Speaker Diarization of Known Speakers

[tobiasschmidt89]

Hi,

I really enjoy using this Library for speaker diarization to create labeled transcripts in combination with Whisper: Speaker 1: ..., Speaker 2: ..., Speaker 1: ...
Currently I then search and replace the anonymous speaker labels with the real names.

I have some meetings that always have the same speakers (D&D game sessions). Therefore I am searching for a way to kind of create "voice embeddings" of each speaker by recording them in isolation for a minute or so. Then I want to do a speaker diarization using these embeddings to get labels like: Max: ..., Maria: ..., Tobi: ..., Max: ..., Unknown: ...

I would be interested if someone has some pointers on how I could achieve this with Pyannote. I expect I need to do the following:

• Use an audio embedding model to create a normalized embedding for each speaker (with an isolated audio file).
• Do a speaker diarization of the entire audio file to get the audio segments.
• Create an audio embedding of each audio segment.
• Calculate the similarity score of each audio segment embedding with the precalculated embedding of each speaker to identify the most likely speaker.
• Use whisper to transcribe each audio segment attaching the label of the speaker.

Crossed items I know how to do.

I am very comfortable with text embeddings. Audio embedding is a new topic for me.
I would really appreciate any pointers or example scripts.

Thank you
T.

[desicochrane]

Hey @tobiasschmidt89 I'm just started working on this too (for podcasts). Digging into pyannote I see it uses speechbrain "speechbrain/spkrec-ecapa-voxceleb" embedding model. https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb

I can share the approach I have in mind:

1. I am probably going to store k representative embeddings per speaker (I'll likely do k-means clustering of al samples of each speaker).
2. Run pyannote diarization. I might provide a high number of speakers, since I'll later be mapping them anyway, I am ok with low precision at this stage to get higher recall
3. Take these and perform dense retrieval against the embedding db. Probably just cosine distance and take, say, 20 nearest neighbours.
4. Then, I'll do a rerank (cross encoder pattern) to get a probability score.

I'll see how that goes.
Likely I can do more here, I have the text of the audio intervals, so I can also compare the text embeddings along with audio embeddings. Also, in my case, I know some stuff about the prior probabilities ahead of time (the hosts of the podcast are most likely. You might have prior knowledge in your meeting use case too).

Let me know how you get on with it. Feel free to hmu if you wanna collaborate a bit on this.

[PhilipAmadasun]

@desicochrane Could you please explain this "Rerank Cross Encoder Pattern" thing? I'm doing something similar to what you guys are trying, might need that. Also, if you take multiple isolated voice samples(embeddings) of the people you know will be in the conversation, will then have to compare cosine distance on every single one of the voice samples you have on one individual person. I ask because let's say you have 10 voice samples per person, that's 100 cosine distance comparisons no?

[desicochrane]

Hey mate, sorry late response.

"Rerank Cross Encoder Pattern" thing?

Checkout the retrieve and rerank pattern here: https://www.sbert.net/examples/applications/retrieve_rerank/README.html

if you take multiple isolated voice samples(embeddings) of the people you know will be in the conversation, will then have to compare cosine distance on every single one of the voice samples you have on one individual person. I ask because let's say you have 10 voice samples per person, that's 100 cosine distance comparisons no?

You could do that, sure. 100 cosine distances is lightning fast and vectorizable. 1M cosine distances can be done in milliseconds.

[PhilipAmadasun]

@desicochrane How can you calculate many cosine distances in real time as you mentioned? Would you happen to have some example code?

[desicochrane]

you could put the 1M vectors into a matrix and do a dot product (if they're normalised), or you could put them in a vector index like Faiss. Maybe something like this:

!python -m pip install faiss-gpu

import faiss
import numpy as np
import time

# Number of vectors and dimensions
num_vectors = 1000000
dimensions = 1000

# Generate random vectors (1M vectors + 1 query vector)
database_vectors = np.random.rand(num_vectors, dimensions).astype('float32')
query_vector = np.random.rand(1, dimensions).astype('float32')

# Normalize the vectors (FAISS requires normalized vectors for cosine similarity via the inner product)
faiss.normalize_L2(database_vectors)
faiss.normalize_L2(query_vector)

# Create a GPU FAISS index (inner product for cosine similarity, as vectors are normalized)
res = faiss.StandardGpuResources()  # Use all GPU resources
index = faiss.IndexFlatIP(dimensions)  # Inner Product (IP) index
gpu_index = faiss.index_cpu_to_gpu(res, 0, index)  # Move index to GPU 0

# Add vectors to the index
gpu_index.add(database_vectors)

# Perform the search
k = 10  # Number of nearest neighbors to find
start_time = time.time()
D, I = gpu_index.search(query_vector, k)  # D = distances, I = indices of the nearest neighbors
end_time = time.time()

print("FAISS-GPU search time:", end_time - start_time)
print("Distances:", D)
print("Indices:", I)

Running that in google colab I got
FAISS-GPU search time: 0.05401325225830078
Distances: [[0.7987112 0.79602313 0.79593 0.7954348 0.79437023 0.7937976
0.7922267 0.7918892 0.79174185 0.79161 ]]
Indices: [[388597 671101 817522 743371 101155 474046 17489 622588 665845 341719]]

[ymednis]

Hi @desicochrane, @tobiasschmidt89 .

I'm exploring ways to improve speaker diarization in recordings with multiple speakers, but my primary goal is to accurately detect a specific target speaker. I have additional enrollment samples for that speaker (which I suppose I can use to generate an embedding), and I'm wondering if anyone here has achieved high precision for this particular use case.
Thanks, in advance.

[tobiasschmidt89]

No, not yet. I am for my use case (D&D Session Recordings) utilizing LLMs for the speaker assignment. It is like a riddle for the LLM. Here are X speakers, here is the transcript and possible speakers. What is the name of each speaker?

[alunkingusw]

Hi, here is my gist that applies embedding on a wav of a known speaker, and compares it to the output of the diarisation. I am building an API that manages meetings and audio from meetings to transcribe it.
Let me know if this is helpful.
https://gist.github.com/alunkingusw/4275a26ba79cefd49f5a2e91d91c4da2

[alunkingusw]

Once I have this working, my API (along with a frontend) will accept audio files and transcribe them, automatically labelling known speakers. The sample audio used for the embeddings could be provided by the user, or manually selected from a recording. Currently I'm looking at how I can take the segments from a meeting and order them by the most accurate detections of speech (https://pyannote.github.io/pyannote-metrics/reference.html), so that the comparisons of embeddings can be more accurate.

[mroconnell]

In case it's helpful, there's a dev who did this process for public meetings in NYC and developed a great prompt engineering methodology to have LLMs identify speakers and then map the speaker name across very lengthy sessions. If you're running something like a 5+ hour D&D session, you might have a lot of 'lost in the middle' problems where the LLM does a much better job IDing speakers at the beginning and end of the file. He has a lot of tricks to fix that. He does mention toward the end that you can include a list of the possible speaker names in the prompt as a way to eliminate small variations and spelling mistakes that occur if you just set the llm loose without that parameter.

Link to his speaker id prompts: https://gist.github.com/voberoi/3d82f6b2a55e79b7cd014847853be8bf
Video presentation on the topic: https://www.youtube.com/watch?v=FjDQu2IbC1M

[mroconnell]

Now that I think about it, if your D&D friends are like mine, they might be voice acting and changing their voices very deliberately as they narrate actions. That would pose an interesting challenge for this specific task! Wonder how that will go.

[voberoi]

That dev is me! So fun to see this here, because I'm investigating creating voiceprints for everyone in my database.

(The site is citymeetings.nyc)

Are Pyannote's service's voiceprints just embeddings? Can I do nearest neighbor searches to find most-alike voices?

[voberoi]

Btw annotated slides for the talk are here if you're interested:

https://vikramoberoi.com/posts/how-citymeetings-nyc-uses-ai-to-make-it-easy-to-navigate-city-council-meetings/

There's a lot of stale info there now. I gave another talk at School of Data in 2025 but it's not up yet for some reason.

[mroconnell]

Are Pyannote's service's voiceprints just embeddings? Can I do nearest neighbor searches to find most-alike voices?
tldr Yes, and I think yes (and it will return scores for each embedding you want it to compare to)

I didn't dive very deeply into this and I'm a real rookie. The test I ran let me upload specific speaker videos so pyannote could create some embeds (I think that was the right language) and then when I ran a whole meeting past it with diarization, it would look at each diarization segment and give a match score for each embed I had (I think this is the NN part of your question) and it would spit out the one with the highest score. I did a mediocre job with the embeds just to see how it all worked and it got 3 out of 5 council members right on my first run through. I was in a hurry to ship something for a demo so I switched it to manual speaker mapping where user just clicks on the "Unknown Speaker 1" label and edits it in the doc. https://www.loom.com/share/dfa4b5c783304d44b53c2cd6cf3109fa

What I intend to do is let users indicate start and end times for uninterrupted speech and upload it to pyannote so the users can create their own embeds for local electeds. But that is a long way away on the road map! :) and naturally you're in a different place where you have a very large but finite list of elected officials you could do a great job for so you might have a smarter path for nyc.

[voberoi]

Interesting! Thank you for the context.

[Nazkab10]

@desicochrane Did you end up figuring this out for podcasts. Looking into the exact same thing rn.

[Nazkab10]

@mroconnell Thanks, taking a look at this