VideoDecoder bit exactness

[sprangerik]

As far as I can tell, the spec today makes no explicit mention about bit-exactness of the output from a VideoDecoder.

A decoder by definition needs to have a bit-exact representation of the reconstructed state in each reference buffer, otherwise there will be video corruption in the form of prediction drift. However, some decoders may perform additional processing on a copy of the reconstructed state, before handing the frame to the user. The post-processing could be in the form of e.g. deblock or debanding algorithms, sharping/superres filters etc etc

I think it would be prudent to:

(1) Add spec language to indicate that the expected output of a decoder instance should be the bit-exact exact form.
(2) Add a field in VideoDecoderConfig that allows post processing to be turned on if supported.

With (1) it becomes trivial to create qualification tests as you can just use a hash the known output for each frame.
With (2) we optionally allow further enhancements to be applied, but only when desired.

It's debatable if (2) should just be an on/off toggle, or if something more fine grained is needed (e.g. a post processing "strength" slider, or even toggles for each kind of available filter).

[dalecurtis]

It's unclear what you mean by bit-exact, since your examples exclude the actual most common reason for mismatch: color space and pixel format. All platforms have preferred formats they want to decode to and doing anything else results in the UA either having to make a copy w/ color space conversion or deal with whatever format came out of the hardware decoder.

At least in Chrome we have instances of such behavior that's inseparable from the decoder API usage on macOS, Android, and Windows. Probably in the Linux path as well, but I'm less familiar with that.

(2) in practice would just prevent the usage of the hardware decoder.

[sprangerik]

What I talk about bit exactness it's most in regards to correctly adhering to the spec. E.g. if you are compliant with the AV1 Bitstream & Decoding Process Specification a given bitstream should result in exactly one set of values for the samples in the reconstructed state. While it's true that color space and pixel formats can affect that, the assumption is that you will be able to get an output in a format where each sample can be mapped to the value in that reference buffer. I.e. "if you decode this bitstream and look at coordinate (x,y) of the chroma plane use color space z you will find the value V" - and that should work for any decoder that adheres to the spec, no?

(2) in practice would just prevent the usage of the hardware decoder.

While many hardware manufacturers implement postprocessing, I don't think that's strictly true. There would e.g. be nothing preventing a nVidia RTX VSR or something along those lines from being implemented in the decode pipeline if those to and there are knobs to enable it, just to name one example.

[dalecurtis]

If your definition of bit-exact excludes color space conversion (rgb <-> yuv, limited/full, etc), then I'd say all browsers are probably already doing what you want. In Chrome at least, VSR isn't applied until the presentation layer since frames may be dropped or not even sent for display.

[sprangerik]

Yeah, it's been my assumption that this is the case. My point is just that it would be great to have that in the spec, so that a user can be sure about the expected behavior. Should be a trivial update.

In Chrome at least, VSR isn't applied until the presentation layer since frames may be dropped or not even sent for display.

Sure, but I think there's opportunities to do things in the decoder itself down the line - especially in terms compensating for coding artifacts since then the processing has the opportunity to know the quantization levels in question, where block boundaries are etc.