How do i fine tune whisper to transcribe the numbers properly

[nagatarunkumar]

I have my own dataset consisting of insurance claim recordings, where users tell various numbers such as policy number, phone number etc and i am using whisper to transcribe these call recordings, Now the issue i am facing is when user speaks a number such as "7998551771" as seven double nine eight double five... instead of converting it to the number as we may expect its converting it into words, So how do i fine tune whisper to basically convert all such instances where user uses words like double, triple while dicating the number to a number instead of words.

[glangford]

Rather than fine tuning Whisper for such a task, it would be easier to post-process a portion of the transcript using regular expressions. Here is how to handle "double" as an example

>>> import re
>>> transcript = "seven double nine eight double five"
>>> re.sub(r"double (\w+)", r"\1 \1", transcript)
'seven nine nine eight five five'
>>>

To then convert words into a number you could use something like this package.

https://github.com/ShailChoksi/text2digits

>>> from text2digits import text2digits
>>> t2d = text2digits.Text2Digits()
>>> t2d.convert("seven nine nine eight five five")
'799855'

And for variations where the caller groups digits, text2digits seems to handle that. For example,

>>> t2d.convert("eighteen eighty-five six zero seven")
'1885607'

[glangford]

Here is a more complete example to demonstrate multiple rules for homonyms, doubles and triples, and digit grouping.

import re
from text2digits import text2digits

test='owe ate double four too triple nine fifty-five'
replacements = [(r"double (\w+)", r"\1 \1"), 
                (r"triple (\w+)", r"\1 \1 \1"),
                (r"oh|owe", r"zero"),
                (r"won", r"one"),
                (r"too", r"two"),
                (r"ate", r"eight")
                ]
for pat, repl in replacements:
    test = re.sub(pat, repl, test)
t2d = text2digits.Text2Digits()
result = t2d.convert(test)
print(result)

0844299955