Reproducible bug in Whisper API regarding segment timing

[towel]

Using the OpenAI Whisper API, I'm getting timings that are partially off and some are predictably completely wrong.

I'm trying to generate subtitles for audio and so far have implemented some hacky workarounds that help me fix the issue only somewhat (transcribing audio per-sentence and re-calculating the time of the last segments), but I don't think I should have to.

Steps to reproduce:

1. transcribe audio file with multiple spoken sentences using Whisper API with response_format verbose_json (haven't tried others)

Expected result:

correct timing info for start and end for all segments, computed length adds up to transcription duration

Actual result:

1. the last segment always has a value for end that is way too long (like 30s for a segment that's actually ~5s).
2. some segments have lengths (ends) that are a bit off. this especially occurs when there are pauses in the transcribed audio, but the "verbose json" doesn't give any information regarding detected pauses to account for this.
3. the accumulated time of the segments (end - start for all segments) doesn't always add up to the reported transcript duration.

Additional information

Using latest openai python API wrappers

[towel]

Using whisper locally, and gaining more understanding of its inner workings, I'm suspecting at least part of the bug described above (1.) is related to audio being manipulated initially via pad_or_trim before being processed in segments of fixed length.

The API should account for this though, imo.

[ryanheise]

1 and 2 are addressed by PR #1114 . Feel free to test it and see if it helps in your case.