Frameskip for depthmaps

[flflow]

How difficult would it be to add something like frameskip for Desktop depth map generation?

For example, with a frameskip 1, the app captures and displays at 60 fps, but the depth map is generated for every second frame instead of every single one (or intermediate depth maps could be created using a lightweight interpolation algorithm).
In theory, there shouldn't be a massive hit to working quality, but it could significantly boost performance, potentially enabling real-time 4K screen streaming and make the tool viable for a much wider range of hardware.

[nagadomi]

When considering performance, the important point is how much of the total cost the current depth estimation accounts for.
Even if frame skipping is used for depth estimation, the improvement is only about 10-20%.
Also, since depth estimation is performed at a fixed resolution, the processing time is not affected by the input resolution. The slowdown caused by higher resolution comes from screen capture, device memory transfer, and image encoding, so the improvement ratio would be even smaller.

[nagadomi]

With #628 and #629, the previously mentioned issues, screen capture, device memory transfer, and rendering (Local Viewer), have been resolved.
As a result, overall performance is now primarily determined by the efficiency of the 3D conversion process.
Depth estimation is still performed at a fixed low resolution, while warping (stereo generation) operates at the original high resolution. Therefore, as the resolution increases, the performance impact shifts toward processes other than depth estimation, although warping relatively lightweight compared to it.

For smooth and visually pleasing frame interpolation, optical flow is required.
However, ML-based optical flow can be as heavy as, or even heavier than, small depth estimation model (for reference, any_v2_s can reach around 160–200 FPS).
Fortunately, recent NVIDIA GPUs feature dedicated optical flow hardware, which could enable frame generation with minimal additional overhead.

If this approach is implemented, it would work as follows:

1. Capture the screen at 60 FPS.
2. Resize the captured frames to a lower resolution and compute the optical flow between every two consecutive frames using Optical Flow Accelerator.
3. Use the computed optical flow to warp (interpolate) the depth images, effectively reducing the depth estimation FPS by half.

In situations where depth estimation can reach 160 FPS, it's probably not the bottleneck in the first place.
However, this approach might be useful when using a large depth estimation model or increasing the depth resolution.