Performance metrics for evaluating time segmentation of Whisper models

[mafaisalpg]

Hello everyone,

I like to compare different Whisper models on my own datasets. My output looks like
start, end, text

Questions

1. What are metrics (MSE, MAE, etc.) best fit for this purpose?
2. Is there any Python implementation which I can reuse?
3. Is there any paper or article have done such comparison?

Thanks

[phineas-pta]

most metrics prioritize word accuracy over timestamps accuracy

but if u want to go down the rabbit hole, the keywords to google/chatgpt are something like "metrics for alignment of time series"

[itaipee]

measuring timing accuracy ,are usually part of diarization evaluation , and the criteria is "diarization error rate" , and can be more complex than evaluate WER.
there is good package about diarization :
https://github.com/pyannote/pyannote-audio?tab=readme-ov-file

they has reference to how they measure their success : https://pyannote.github.io/pyannote-metrics/reference.html

[mafaisalpg]

thanks. however, I am struggling to change my vad annotation file (start, end, ) to pyannote-audio compatible one for "Detection Error Rate". any suggestion on this?