GPU -> CPU fallback: difference and alignment between the BETA and default interfaces #943
Description
When it's not capable of decoding a video with NVDEC, the default interface (from FFmpeg) falls back to the CPU decoding path. This can happen in various scenarios, all related to a codec configuration that isn't supported by NVDEC:
- when decoding a 10 bit video NVDEC (for example)
- when decoding a video that's too small, which isn't supported by NVDEC (example)
In the BETA interface, however, we don't fall back to the CPU. We error loudly.
Below are just my current thoughts, this is obviously open for discussion.
I think we will want to fallback on the CPU for both default and BETA interfaces. Otherwise, we're essentially putting the fallback responsibility on the user, which isn't a good UX. The user wants the video to be decoded. They don't want to deal with errors.
I also think we'll want to fallback on the CPU only for the decoding part. Currently, when we fallback on the CPU, we also perform the color-conversion stage (YUV -> RGB) on the CPU. But it would make sense to do the color-conversion on the GPU. Presumably, it'd be faster. And it would also optimize the CPU -> memory transfer, since the YUV frame is smaller than the RGB frame.
Both interfaces should have the exact same behavior: both to fallback on the CPU, and run the color conversion on the GPU. For both interfaces, this is a somewhat BC-breaking behavior change. But I don't think anyone would complain.