Whisper API - word level segments?

[bytewiz]

I am curious to understand if there are any plans for the whisper API to support word-level transcription. Or if it is somehow possible to infer it from the verbose response with the tokens array which I assume is just to show the tokens spend for each segment?

Getting word-level segments would give much more flexibility.

(If I am asking the wrong place, sorry, do let me know where to ask otherwise)

✌️

[ryanheise]

This is available experimentally by using the --word_timestamps True option on the command line and the json output format. Whisper will have trouble identifying exactly how to split up words in languages that don't separate words by spaces (e.g. Chinese and Japanese) in which case the "words" you get back might be at a level of granularity a bit smaller than an actual word on average, but it should work correctly for languages such as English which do separate words by spaces.

[bytewiz]

@ryanheise nice! will it come to the API too at some point? if you know?

[ryanheise]

I'm not personally familiar with the API, although it appears there is a separate forum for that product:

https://community.openai.com/tag/whisper

[glangford]

Reading a json with word level timestamps as @ryanheise suggests works well (in English) so far and the timing accuracy has been good. It is possible to use NLP to realign speech into sentences and phrases (for example) and realign timing for subtitles. I am using spaCy experimentally for this purpose.

[jon-jett]

Can you say more about this? I looked at it but decided spaCy was mostly about sentiment extraction and I didn't know how to make it break sentences naturally. I wrote a thing to limit word/character length per line but also distribute words among subtitles so you don't get one subtitle at the end with only one word, etc. ... but it seems spaCy would be more elegant.

[glangford]

Hi @jon-jett , sentence segmentation is built into spaCy, see for example

Sentence Segmentation
https://notebook.community/rishuatgithub/MLPy/nlp/UPDATED_NLP_COURSE/02-Parts-of-Speech-Tagging/04-Sentence-Segmentation

[erdzan12]

Does the OpenAI Whisper API https://platform.openai.com/docs/guides/speech-to-text/speech-to-text-beta support word_timestamps?