Problem to reproduce Earnings21 results

[naymaraq]

In the paper, the authors reported the results on the Earnings21 dataset, reporting a 9.72% Word Error Rate (WER). However, upon attempting to replicate these results, I encountered some difficulties.

Could you please provide me with more information on your normalization process?
Specifically, do you normalize both the output files and the golden files?
Additionally, I'm curious to know whether you used the fstalign tool provided in the Earnings21 benchmark.

[jongwook]

Yes, we applied the normalizer to both the ground truth (reference) and the predictions (hypotheses). This makes the WER numbers not directly comparable with the literature. We haven't used the fstalign tool.

The benchmark numbers included some nondeterministic settings such as the temperature fallback, so it might be difficult to exactly replicate the WER, but it should probably land in the ballpark.