Improved Highlighting Animation

[griff12]

I would love to see a word by word highlighting animation that works for all file types. The Eleven Reader app implementation looks quite nice in this regard.

[richardr1126]

This is especially difficult because I have chosen to support multiple TTS apis and most do not support word by word timestamps in the audio as far as I'm aware

[griff12]

I believe that Kokoro does. From the Kokoro-FastAPI page:

Timestamped Captions & Phonemes
Generate audio with word-level timestamps without streaming:

import requests
import base64
import json

response = requests.post(
"http://localhost:8880/dev/captioned_speech",
json={
"model": "kokoro",
"input": "Hello world!",
"voice": "af_bella",
"speed": 1.0,
"response_format": "mp3",
"stream": False,
},
stream=False
)

with open("output.mp3","wb") as f:

audio_json=json.loads(response.content)

# Decode base 64 stream to bytes
chunk_audio=base64.b64decode(audio_json["audio"].encode("utf-8"))

# Process streaming chunks
f.write(chunk_audio)

# Print word level timestamps
print(audio_json["timestamps"])

Generate audio with word-level timestamps with streaming:

import requests
import base64
import json

response = requests.post(
"http://localhost:8880/dev/captioned_speech",
json={
"model": "kokoro",
"input": "Hello world!",
"voice": "af_bella",
"speed": 1.0,
"response_format": "mp3",
"stream": True,
},
stream=True
)

f=open("output.mp3","wb")
for chunk in response.iter_lines(decode_unicode=True):
if chunk:
chunk_json=json.loads(chunk)

    # Decode base 64 stream to bytes
    chunk_audio=base64.b64decode(chunk_json["audio"].encode("utf-8"))
    
    # Process streaming chunks
    f.write(chunk_audio)
    
    # Print word level timestamps
    print(chunk_json["timestamps"])