How to automatically generate subtitles for video and translate them for NextJS

[CareNetAI]

We want to do the following:

• User logging in to our website
• He click "upload video" and choose the video from the disc
• Video is uploaded to cloud
• Once it's uploaded, the speech from video (no matter what langueage it is) should be converted to text and then translated to english.
• File with captions should be saved so that user can pick them from the "Subtitles" menu in video player.

The tricky point is 4) because I don't know which software could I use to generate subtitles from video and translate them to english (and possibly other languages). I found Whisper AI from OpenAI and I'm wondering whether I should use this one or something else. Also, it's a Python library and I'm wondering how can I actually call this when upload finished. I'm using NextJS

[Mattral]

I think There is a package written in nodejs at: [https://www.npmjs.com/package/openai]

for example:

import fs from "fs";
import OpenAI from "openai";
const openai = new OpenAI();

async function main() {
  const transcription = await openai.audio.transcriptions.create({
    file: fs.createReadStream("audio.mp3"), // formats: flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, or webm.
    model: "whisper-1",
  });

  console.log(transcription.text);
}
main();

[CareNetAI]

Thanks

[matusnovak]

Not quite right. This won't generate timestamps (for subtitles, as the OP asked). You must enable that explicitly, see here: https://platform.openai.com/docs/guides/speech-to-text/timestamps

[satishSingh0710]

You can use creatomate for that. It's very powerful and can be customized.