Inconsistent wer on malay language compared to result published #1973
Unanswered
josephwong14wkh
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I am currently evaluating the performance of transcription on malay language (FLEUR testing dataset). However, my result is not the same as the published result. Here is the wer i got:
model size: wer:
medium 13.218%
large-v2 9.497%
Here is the published wer:
model size: wer:
medium 12.2%
large-v2 8.7%
Here is my script:
from jiwer import wer
from whisper.normalizers import BasicTextNormalizer
def cal_wer(model_size):
df = pd.read_csv(gtfile, header=None, sep="\t")
model = whisper.load_model(model_size)
gt_texts = list()
pred_texts = list()
I use https://github.com/jitsi/jiwer to evaluate the wer between ground truth text and STT text and use text normalizer provided in whisper/normalizers/basic.py. May i know is there any configuration (properly on DecodingOption?) that i need to set so that i can replicate the published result?
Beta Was this translation helpful? Give feedback.
All reactions