Use whisperx and pyannote in Colab without HuggingFace token

[biagioscalingipsy]

Hello! I would like to use WhisperX and Pyannote as described on this GitHub to combine automatic transcription and diarization. I can do it on Colab using the Huggingface (HF) token, but I would like to avoid entering the HF token every time. So I was thinking of downloading them locally and loading them when needed. I can do this for WhisperX but not for Pyannote. Therefore, I can do the automatic transcription but cannot proceed with the annotation. I followed these tutorials on how to use Pyannote and the offline pipelines and downloaded everything, but it still doesn't work. Can you help me?

I download pytorch_model.bin and configuration.yaml for voice_activity_detection, then yaml files for segmentation and speaker_diarization and put them in working directory. Then I used this code:

!pip install whisperx
import whisperx
import gc

device = "cuda"
batch_size = 4 # reduce if low on GPU mem
compute_type = "float16" # change to "int8" if low on GPU mem (may reduce accuracy)

audio_file = "audio.wav"
audio = whisperx.load_audio(audio_file)
device = "cuda"
compute_type = "float16" # change to "int8" if low on GPU mem (may reduce accuracy)
batch_size= 16 # reduce if low on GPU mem
model_name = "large-v2"

model_dir = "/content/drive/MyDrive/whisperx_models/large-v2"
model = whisperx.load_model(model_name, device, compute_type=compute_type, download_root=model_dir)

audio = whisperx.load_audio(audio_file)
result = model.transcribe(audio, batch_size=batch_size)
print(result["segments"]) # before alignment

model_a, metadata = whisperx.load_align_model(language_code=result["language"], device=device)
result = whisperx.align(result["segments"], model_a, metadata, audio, device, return_char_alignments=False)

print(result["segments"]) # after alignment

!pip install pyannote.audio

pipeline = Pipeline.from_pretrained("/content/drive/MyDrive/speaker_diarization.yaml")

# Apply diarization
diarization_result = pipeline(audio_file)

[FrenchKrab]

I'm not really familiar with colab, but it probably still downloads and store the .bin model in some temporary file cache instead of your drive.
What you can do is edit the speaker_diarization.yaml and change segmentation: pyannote/segmentation-3.0 with segmentation: /content/drive/MyDrive/pytorch_model.bin so that the 'segmentation' model uses your own downloaded version.

[biagioscalingipsy]

Thank you for the reply!
I download bid model and yaml file and modified the path to the model:

from pyannote.audio import Model
offline_model = Model.from_pretrained("/content/drive/MyDrive/Speech_to_Text_Diarization/pytorch_model.bin")

from pyannote.audio import Pipeline

audio_in_memory="/content/drive/MyDrive/file.WAV"
offline_speaker = Pipeline.from_pretrained("/content/drive/MyDrive/Speech_to_Text_Diarization/speaker_diarization.yaml")

Yet, when i try to apply the pipeline to the audio by using:

offline_speaker(audio_in_memory)

I get this error:

"RuntimeError: A pipeline must be instantiated with pipeline.instantiate(parameters) before it can be applied."

This is how the pipeline is applied in pyannote tutorial:

offline_vad = Pipeline.from_pretrained("config.yaml")
offline_vad(audio_in_memory)

How can I fix this error?

[FrenchKrab]

The error message lacks details, but a common cause for this error is using a 'multilabel' segmentation model instead of a 'powerset multiclass' model.

Maybe your current pipeline yaml relied on a powerset multiclass model (does not need a segmentation threshold, as we take the argmax of the multiclass probabilities to get the discrete final prediction), but you changed it (the one you downloaded) to a multilabel-based segmentation model (needs a segmentation threshold to discretize/binarize the individual speaker activations in [0;1]).

You can check the output type of the model with model.specifications.powerset (true or not).

If that's what's happening, you can either add threshold: 0.5 under segmentation: in your pipeline yaml, or use a powerset model to fix the problem.

[biagioscalingipsy]

I wrote offline_model.specifications.powerset and the output is "False". So i modified the yaml content by adding "threshold" here:

"params:
clustering:
method: centroid
min_cluster_size: 12
threshold: 0.7045654963945799
segmentation:
min_duration_off: 0.0
threshold: 0.5"

Then I am using this: offline_speaker(audio_in_memory) and it's taking a while but it is running, so I think it works.
Do you know there is any way to make the process faster?

[FrenchKrab]

In order of importance

1. make sure it is on gpu pipeline.to(torch.device("cuda"))
2. use as batch sizes as large as your gpu allows (eg pipeline.embedding_batch_size = 16 and pipeline.segmentation_batch_size = 16)
3. also apparently the suggestions in Pyannote is 10 times slower than WhisperX with GPU utilization 10%: expected behavior or misconfiguration? #1652 speed up the inference for some people ? Still not sure about what is this issue but you can try