Can I use Whisper to output a set of phonemes with timings?

[blackears]

I was thinking that this might be a good tool to automatically create lip sync from audio tracks. For output, though, instead of English words I'd need a list of basic phonemes and the time they occurred measured in milliseconds. Is Whisper able to do this?

[phineas-pta]

word timestamp YES
phoneme timestamp NO

[blackears]

I found the whisper_timestamp library, which is good for getting word timestamps. I don't see a way to do it with just whisper, though.

Also, I had thought that phonemes would be even simpler than words, but another thread I read made it seem like it would be more difficult. If there is any library for parsing phonemes, I'd like to know about it, but if this sort of technology doesn't exist yet, it would be interesting to know why words are easier than phonemes.

[phineas-pta]

in what way phoneme is easier than word ?

[blackears]

There are fewer phonemes and they make up smaller units. Phonemes to spoken words are like letters to written words.

[dgoryeo]

@blackears , I'm guessing you've already looked into WhisperX? They use a phonem based model in parallel with Whisper to adjust timestamps. Perhaps their approach can help your case too?

[blackears]

I did, but it requires CUDA to work and I want something that can work without that.

[marceltud]

Hi,

Do you get phonems out of WhisperX? I thought you get a timestamp for each word, but not a transcription into phonems, can you please confirm that?