exploring multi-format audio and video integration

[devmanbr]

I stumbled upon this project by chance and already fell in love with it. Thank you so much for developing it, @Vanilagy!

I have a priority project in my team that's been on hold for a while because I couldn't implement it in electron.js. It involves the ability to play video and audio in various formats, supporting multiple audio tracks and subtitles (as we've seen in mkv, avi, etc.), which electron.js unfortunately does not support natively. ffmpeg was ruled out due to the excessive overhead it caused during the tests I conducted, as there is a strong requirement that computers with modest configurations do not experience playback issues.

I’m getting familiar with Mediabunny, and it really seems to be the solution to all our problems, but I haven't had the time to test the implementation for the use case I need to cover. Since that would take some time, I thought it would be more convenient to discuss this beforehand.

I’m opening this discussion to exchange ideas with anyone who has become familiar enough with the library, so that I (and others in the future) can better understand the capabilities and any potential limitations we may face.

thank you all in advance!

[Vanilagy]

Lovely! Mediabunny is still new so doesn't have support for all container formats out there (although that's also not a goal), but it's exactly the right foundation for what you're trying to build!

[absidue]

ffmpeg was ruled out due to the excessive overhead it caused during the tests I conducted, as there is a strong requirement that computers with modest configurations do not experience playback issues

Were you using the latest version of ffmpeg and did you pass the correct flags to enable hardware acceleration for the hardware you were using? If you can't get decent playback even with a correctly configured ffmpeg, which is written in compile to native languages and assembly, I would be very surprised if you could even get matching performance with something written in JavaScript, never mind better.

[Vanilagy]

Mediabunny interfaces with the WebCodecs API, which is written in compile to native languages and also interfaces with hardware accelerators. You can reach very impressive speeds with that. FFmpeg, when writing client-side applications, is just a big pain to interface with. Especially FFmpeg WASM, whose performance often just sucks.

[devmanbr]

It's basically what @Vanilagy said. In all my tests, the results encouraged me not to use ffmpeg.

[devmanbr]

Once again, Mediabunny came up in a discussion with my colleagues due to a customer's request to develop a command line tool for creating and transforming videos in various formats.

I'm loving to imagine the possibilities I will have with the tool.

Although I really don't have experience with codecs, I would love to collaborate on the project in some way.

P.S.: I apologize for the long period of silence in the discussion. I had to step away from work to take care of my health, but I'm slowly getting back into it.

[Vanilagy]

If you guys want a command line tool, wouldn't FFmpeg be the obvious choice?

[devmanbr]

Generally, yes. But I like the idea of ​​building an SDK or something similar underneath beforehand, to facilitate eventual porting to other platforms/environments in the future.

Having something with the capability to be highly pluggable would greatly facilitate the work of developing the SDK.

As I said before, ffmpeg doesn't work well for these cases.

I'm evaluating to what extent Mediabunny could be "portable" to environments other than browsers. If it's at least easier and/or more performant than ffmpeg, that's already a big plus.

[Vanilagy]

Porting it to other environments is possible, but it requires polyfilling the codecs, which is non-trivial. Although I did develop webcodecs-polyfill to help with this.

[devmanbr]

Definitely, working with audio/video is not trivial, and certainly the webcodecs-polyfill package will be of great help in this process, from what I understand.

I’m not sure if this is part of your vision for the project, but it would be a very positive point if we could pave the way for Mediabunny to be used in any environment/platform.

[Vanilagy]

It runs everywhere JavaScript runs, and you can add support for encoders/decoders using the custom coder API that Mediabunny provides. One major limit right now is that Mediabunny also makes use of the canvas API, which right now has to be polyfilled for it to work. (most code paths don't use it ofc, it's used for resizing video)