Is there "seek" functionality for Videos?

[pie6k]

I am creating a class that takes video url and has a method .seek(timeMs) which returns VideoFrame promise and wondered:

• is there something like this built in
• what's the best and most optimal method of doing it

The primary goal of this is performance and time you have to wait

The class needs to be stateful, examples:

• if I seek for 2s and then instantly after for 2.1s, we should not look for new keyframe, but instead keep consuming the stream we already have as it will be faster and it is likely the frame we need is just after the one we returned a moment ago (thus we should also keep our VideoDecoder instance)
• however, if we did seek for 2s and now seek for 10s, there is no point in consuming our current stream with existing decoder, we probably need the closest frame before 10s and start from there. However - in this case, eg if video is 60fps and last keyframe was at 9.1s, we will likely need to throw away around 50 frames before we get the one we need, which might be slow - is there a way to speed it up?

In general it should behave like a normal html Video that aims to show a new frame as quickly as possible after you seek it, in whatever way you seek it (looping, quick mouse drag over timeline in any direction, etc)

I am ok, trying to implement it myself, but some guidance about the most performant (and memory efficient) way of doing that would be useful

[Vanilagy]

This is an issue that many people have run into already and a missing hole in Mediabunny v1's API. Fear not! Mediabunny v2 already has a solution to this in the form of SampleCursor, which is a stateful object with methods like seek and next, optimized for optimal decoder usage and highest performance. In Mediabunny v1, this doesn't exist yet, but you can kinda build it with two things:

• VideoSample iterator acquired with VideoSampleSink.samples(startTimestamp); this does the decoding, keep it around for consuming the "next" frames.
• An EncodedPacketSink to aid in key frame probing; basically, when seeking to a new spot, see if the key frame at that spot is earlier than the current timestamp of the VideoSample iterator. If so, there's no point in creating a new decoder, but it's simply smarter to "pull" on the iterator until reaching the desired frame. If the key frame is indeed ahead, it's faster to close the old iterator using .return() and setting up a new one. Obviously, if you seek backwards, you'll always need to set up a new iterator.

[pie6k]

Wow. I was not aware of all those v2 things. It looks very impressive. It seems there is no v2@beta on npm, so it is probably not testable yet.

One suggestion on this:

"Obviously, if you seek backwards, you'll always need to set up a new iterator."

I would reconsider this with the goal of "show any feedback to the user ASAP" in mind:

• it could be a good idea to keep some buffer of, e.g., 10 prev frames, or even 1 prev frame. For example, if in a video editor the user is moving back frame by frame with the left arrow, this 1 past frame buffer would make a whole difference in providing an 'instant result feel,' i.e., when the user thinks, "Do I want to press the left arrow again?" the previous frame is already ready every time
• If moving backwards by large chunks, maybe it could be worth it to first provide some "almost great" result, e.g., the last keyframe, and then provide an accurate result again when we iterate from the keyframe to the exact frame we need. In terms of UX, while the first frame is not exactly the one you want, it is very close to what you want and will be ready sooner, which already provides some valuable and almost instant feedback to the user. Perhaps the method that returns the final promise could also accept a callback argument 'onPrelimitaryResult'

I can imagine those are quite opinionated scenarios (e.g., compromising accuracy for speed) so it might be out of scope, but in my case, I will do everything possible to provide any feedback ASAP, so I will likely still have to implement it myself, but I am leaving it as a loose suggestion.

[Vanilagy]

Yeah, there's no v2 beta on npm or anything since it's not really done yet, although I'm actually planning on releasing alpha versions to npm for people to try them out.

And I totally know what you're talking about! Backwards seeking without any optimizations provides quite a painful user experience. The north star goal is to provide a UX where the user almost doesn't notice that seeking is taking place; where the video just instantly updates no matter where the user seeks. Generally speaking, keeping a few frames of buffer around the current playback head is a good idea, although keeping full video frames around is limited. VRAM usage is high, and it actually clogs up the underlying VideoDecoder and prevents it from emitting more frames. A better approach is to render the frames to a lower-resolution canvas and store those instead.

Regarding the "almost great" result, I'm also with you! I really need people to test the new SampleCursor API in alpha before it's officially released exactly to see if it is able to model these situations flexibly enough. It does have a seekToKey method, so when doing a large seek, you can always first call seekToKey, followed directly by seekTo, both with the same timestamp. Then you'll get two promise events: One for the key frame, which is almost immediately resolved, and one for the final frame.

An alternative additional API would be something like what you suggested, where you're able to "intercept" all of the intermediate frames the decoder is churning through under the hood before the target frame is reached. Again, need to see what works best.

[pie6k]

Thank you! Sounds great!

Also question about one more use case: with seek, what happens if there is a new seek before the previous one resolves?

It is quite common, eg user moves the mouse over the timeline - it will emit 100 events (and requests) in a second or two

[Vanilagy]

I assume you're talking about the new API. There, calls are queued up and serialized. It's very powerful because the decoder uses that info to "plan" better.

There is an "is idle" method you can call to check if there are still unresolved method calls, so you can choose if you want to spam seeks or only have one at a time.