make edit lists behaviour consistent with ffmpeg

[tonybart1337]

I was running a few tests with edit-list videos with negative pts and I noticed a mismatch in pts values between mediabunny and ffmpeg.

I checked ffmpeg's source code and here's what I found:
https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/mov.c#L4526-L4538

They normalize the start time to 0, while mediabunny simply applies edit list adjustments.

Would you consider sticking to ffmpeg's behaviour?

This is important for accurate frame seeking in different environments to match timelines, for example: mediabunny and pyav.
Mediabunny would report one pts while pyav would use another. While they would seek just fine in their own timelines, they might output different frames if you use the same pts between them

Video example attached for convenience: favtrip_gas_station.zip

[Vanilagy]

I actually think Mediabunny's behavior basically matches that of FFmpeg here. If you compare the video packets, there are the start times:

Video:
FFmpeg: -3.712042 seconds
Mediabunny: -3.7054583333333335 seconds (I actually think this is the correct value)

Audio:
FFmpeg: -0.000408 seconds
Mediabunny: -0.00040816326 seconds

FFmpeg values are extracted via ffprobe.

So besides a 0.007 difference in the video track (which I'm looking into), they both match up. Am I missing something? Is it this 0.007 difference that's the problem?

[Vanilagy]

Alright, I think that's what you're talking about! My demuxer produces the first fully-presented frame at PTS 158 (timescale 24000), and FFmpeg shifts that back to 0.

Now the big question: is my behavior wrong? The edit list, by how it is specified, is quite clear:

The edit in the video does not fall on a video sample time, and per spec, that's fine. Any player should take care to make sure the first frame is displayed for a shorter duration than the subsequent frames. Mediabunny deals with this just fine as well. So, I'd argue Mediabunny is doing the correct thing in this case.

I see the appeal for matching FFmpeg's behavior, and I strive to do that in many cases for verifying my own logic, but mimicking FFmpeg's behavior has never been an explicit goal for Mediabunny.

What do you think?

[tonybart1337]

my bad for not specifying it more clearly, yes I meant PTS and 158 tick difference
I never said that mediabunny is doing something wrong or incorrect, on the contrary, mediabunny outputs all the frames as expected in its own timeline

my only concern is that ffmpeg normalizes PTS to 0 for some reason and thus creates a different timeline with different PTS values. Which means that in order to accurately get the same frame in 2 different tools: ffmpeg and mediabunny I would have to use 2 different PTS values

But I want consistency and frame accuracy, if I seek by a certain PTS then I would expect to get the same frame at all times in ffmpeg and mediabunny.

To achieve that I could use ffmpeg timeline and add a wrapper on my side before calling mediabunny's seek function and any other function that require seeking

I think it's possible to craft a video in such a way that when you normalize it to 0 you would get different frames using the same PTS values, you just need to adjust frame duration and timescale to achieve that - this is in theory, I might be wrong

so, my question is should mediabunny normalize to 0 just like in ffmpeg or stick to the current behaviour?

it's also possible to make it configurable to support both cases, but does it belong to the core mediabunny or not, I don't know
I guess the people who use mediabunny require more control over their data than a simple HTML player, so they're probably already using ffmpeg in their pipelines

[tonybart1337]

@Vanilagy please let me know your thoughts:
should I submit a PR for normalization option?

in case you don't think that it belongs in mediabunny, I completely understand and I can close this issue
maybe it can be implemented as a plugin, but I'm not sure if there's an api for that

TBH: I think plugin is the best solution here, because it only covers edit-list in mpeg container, I'm sure there are other cases where ffmpeg normalizes or behaves differently for certain reasons, and for that we might as well create mediabunny/ffmpeg-compat plugin which will cover such cases fully

for now I only stumbled upon edit-list normalization and I think I can handle it on my side