Solution for playing hardware decoded video without copy data from host memory/GPU memory to GPU memory/host memory

[asuper0]

First of all, I want to creat a native window by slint and keep it in the slint ui.

I used to decode video by ffmpeg, which support DXVA2 hardwave acceleration. The output format is a LPDIRECT3DSURFACE9, which is a pointer to a surface. I could render it to a window or control directly by invoking IDirect3DDevice9::Present()

    STDMETHOD(Present)(THIS_ CONST RECT* pSourceRect,CONST RECT* pDestRect,HWND hDestWindowOverride,CONST RGNDATA* pDirtyRegion) PURE;

The para hDestWindowOverride is a native window handle in Windows, With this solution, the data is always kept in GPU memory.
I'm not familiar with Linux, but it seems that Linux has similar window handle. Maybe the video could be rendered like the same approach.
I have read #977 . Slint could use OpenGL texture, but could all the list hardware decode ouput format could be convert to a OpenGL texture?

// hwcontext.h in libavutil
enum AVHWDeviceType {
    AV_HWDEVICE_TYPE_NONE,
    AV_HWDEVICE_TYPE_VDPAU,
    AV_HWDEVICE_TYPE_CUDA,
    AV_HWDEVICE_TYPE_VAAPI,
    AV_HWDEVICE_TYPE_DXVA2,
    AV_HWDEVICE_TYPE_QSV,
    AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
    AV_HWDEVICE_TYPE_D3D11VA,
    AV_HWDEVICE_TYPE_DRM,
    AV_HWDEVICE_TYPE_OPENCL,
    AV_HWDEVICE_TYPE_MEDIACODEC,
    AV_HWDEVICE_TYPE_VULKAN,
};

So, I suggest to add the ability to creat a native window by slint and keep it in the slint ui, so users could use the API of each decode platform the render their texture to the window.

[asuper0]

@tronical @ogoffart do you have any idea?

[tronical]

Embedding a native window is something that we've discussed before and that would seem to be in scope for Slint.

Regarding texture sharing, there's one combination that might work on Windows:

• ffmpeg's AV_HWDEVICE_TYPE_D3D11VA renders into a ID3D11Texture2D
• If you use Slint with ANGLE, then you can import this texture into OpenGL: https://github.com/microsoft/angle/wiki/Interop-with-other-DirectX-code#demo

The question is: How do you use Slint with Angle? The answer is to build angle and place libEGL.dll and libGLESv2.dll` are in the library search path.

I've tried some of our examples with Angle on Windows and that was positive, but I've never tried the combination of angle with d3d11 backend, ffmpeg with d3d11 backend, and sharing textures. I'd be very interested in this if you manage to get it working :)

[asuper0]

I used to decode video by ffmpeg, with the hardware acceleration platform D3D11/DVXA2. Both of the two can give raw texture ID3D11Texture2D/LPDIRECT3DSURFACE9.
As metioned above, I could use IDirect3DDevice9::Present() to render to a native window for the DXVA2 situation.
D3D11, however, I failed to use it to render the texture, with any way I tried.

If you use Slint with ANGLE, then you can import this texture into OpenGL: https://github.com/microsoft/angle/wiki/Interop-with-other-DirectX-code#demo

This may work, but it's a little more complicated. I will try if the native window can't be implement.