Extending Davinci Resolve with Whisper for film editing

[octimot]

I've coded a quick implementation of Whisper for Davinci Resolve, which basically renders out the current timeline and then re-imports the SRT when the transcription/translation is finished: https://github.com/octimot/StoryToolkitAI

We've been using this for a few days now in our film editing workflow and it works significantly better than any other Speech-to-Text solution out there.

A first next step would be to find a way to merge back the segments in such a way that we get the transcripts split into sub-ideas. Speaker diarization would also be nice.

On the longer term, implementing GPT-3 and CLIP into the tool for a bunch of other things to help editors would be really interesting.

Looking for ideas and contributors!

[crazy4pi314]

Oh this is soooo cool!! I am a noob at using Davinci and have really wanted good accessibility options for my videos <3 Will take a look at the repo

[octimot]

Thank you! The tool is currently super raw but we're fixing, cleaning, adding features and updating almost every day. Looking forward for your take on it!

[elementalcollision]

this is great. Typically I use SimonSays.ai to generate the SRT files and then import them to the timelines as required. in-situ processing would help dramatically. Happy to test on my MBP if that's needed (will need some explicit instructions around this, though).

[octimot]

@elementalcollision

Hey there!

Check out the detailed installation instructions here and feel free to get in touch if you have problems somewhere along the way.

I've recently added sentence transformers to do semantic searches in transcripts and it's pretty cool.

Cheers!

[aylardananil]

Hey! Can you take a video of the detailed installation of this for Davinci Resolve?

[nhaouari]

Thank you very much for project.
One idea that I would like to share is to locally being able to edit video as text (same as online services like Descript or Typestudio).

[octimot]

Thanks for the feedback!

I think we need to first make the transcriptions work really really well and integrate it better with Resolve, and then think about text-only editing features (which would be no doubt useful for a lot of people).

[BaGRoS]

I tested, and I like it :D

BaGRoS

[octimot]

Thanks for all the feedback!

I just pushed an update. The model and GPU/CPU selection via GUI are coming soon too!!

[josephnilo]

This looks cool, going to check it out!

I am here to see if I can create a standalone app or plugin for Premiere Pro that will take a text prompt (like the script for a commercial video) and transcribe the audio to SRT using the script as a guide.

The Premiere Pro transcription is OK but then you have to go back through and edit brand names, URLs, grammar / punctuation, etc. This is a pain if you're churning out a lot of content. I'd like an accurate transcription in one pass.

[nofi-sys]

Great Idea! Thank you!

[layercomp]

I love it! It would be really really cool to have an option to limit the number of characters/seconds per range, like Premiere does

[elementalcollision]

Thanks, Octavian. I’ll give it a go a bit later!

…

On Wed, Nov 9, 2022 at 12:24 PM Octavian Mot ***@***.***> wrote: @elementalcollision <https://github.com/elementalcollision> Hey there! Check out the detailed installation instructions here <https://github.com/octimot/StoryToolkitAI/blob/main/INSTALLATION.md> and feel free to get in touch if you have problems somewhere along the way. I've recently added sentence transformers to do semantic searches in transcripts and it's pretty cool. Cheers! — Reply to this email directly, view it on GitHub <#226 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOFRTTNUKFEP3RBSQFJ3SGDWHPM35ANCNFSM6AAAAAAQ2YQQKE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[carlosalosno]

Amazing projetc, thansk for sharing, guess if you take a look on thsi othe rproyect coudl be nice for speaker diarization

#571

So you can identify the speakers too

[octimot]

This will be taken care of in a future update!

[jkf87]

awsome, i used movie.py to make auto-edit video files,
but Sometimes I don't like the results. really really I'm really looking forward to it

[taonergy-bit]

I think it's even more remarkable if it can transcribe and detect the timestamp and you can edit the video just by editing text. Like Descript, but in Davinci Resolve

[octimot]

This can surely be created and extended further, but our main goal for now is to create a great editing assistant using AI - an assistant that knows exactly what is where and can help you find content easily in your footage.

But feel free to open this request on the project issues page.

[RandomNoSkin]

Hey!

I would really enjoy testing/using this script...
But I do not have installed DaVinci Resolve at the default location :/

Tried to have a look into the code and fix it but wasn't able to do so :(

Some Infos:
The "DaVinciResolveScript.py" is at the default location.
My "Resolve.exe" is at "Z:/DavinciResolve"

Would be happy if you could help me!

Philipp aka RadnomNoSkin

[octimot]

Hey there!

Unfortunately the Resolve API cannot be accessed unless Resolve is at the default location. We have no idea how to make it work since the API code is mostly closed by BM.

One way to go around this might be to create a hard link in the Program Files folder to your installation folder via command line. Not sure if it works, but you could give it a try.

Cheers

[RandomNoSkin]

Sooo

After a few attempts i just copied DaVinci Resolve back to the default location.

Works fine now!
Thanks for the work!

[ahujaravish]

Thanks for making this tool. I think it will be very useful. Do you know if this works with the App Store downloaded Davinci Resolve Studio? I tried the instructions and don't seem to find the "transcribe" button in Timeline Menu. Any possible screenshots of how it looks? Thanks again.

[ChristopherSchacht]

@octimot arbeitest du immer noch dran? Das Tool ist so unfassbar hilfreich! Kann kaum erwarten, dass neue Updates kommen! =)

Vor allem für ein paar Bug-Fixes und einfachere User-Zugänglichkeit wäre ich mega dankbar! Bei langen Timelines hängt sich das Tool z.B. beim ersten Mal transkribieren bei mir auf. Wenn ich aber erstmal nur 10 Sekunden mache und dann "Re-Transribe" dann klappen auch 3-4 Stunden footage ohne Probleme. (Auf dem M1 Macbook)
Wäre super stark, wenn sowas in Zukunft auch ohne solche kleinen Workarounds reibungslos gehen würde!

Auch die Installation mit einem "one-Click" funktioniert meist noch nicht einwandfrei. (Die Standalone Versionen funktionieren bei mir bisher nur, wenn ich danach doch nochmal alles selber installiert habe).

So, wie es ist, ist es schon ne enorme Hilfe! Ich meine nur, dass diese "Kleinigkeiten" vermutlich den Unterschied zur Massen-Adoption machen würden.
Hast du Mal BlackMagic kontaktiert und die gefragt, ob du nicht auf Dienstleistungs-Basis das mit nen paar Leuten von denen direkt in Resolve rein implementieren dürftest?
Kann mir gut vorstellen, dass Blackmagic Geld dafür zahlen würde!