toCUDA() slower than regular VideoCapture

[David-rn]

I would like to thank you for such amazing library!

I came accross some unexpected behaviour. Just to add some context and verify that ffmpeg is compiled with cuda support.

The following command give me ~500 fps

 ffmpeg -y -vsync 0  -i input.mp4 -f null /dev/null

And this one give me ~700 fps. And I can see that the gpu is being used by the process.

 ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 -f null /dev/null

I was performing some benchmark with these two examples:

import ffmpegcv
cap = ffmpegcv.VideoCapture(file)
while True:
    ret, frame = cap.read()
    if not ret:
        break
    pass
cap.release()

########################

import ffmpegcv
cap = ffmpegcv.toCUDA(ffmpegcv.VideoCaptureNV(file, pix_fmt='nv12'))
while True:
    ret, frame = cap.read()
    if not ret:
        break
    pass
cap.release()

Nevertheless, the first example complete reading the video in 8 seconds while the second one completed it in ~20 seconds. I was wondering why this could be happening.

I performed some profiling and these are the results for cpu and toCUDA:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.003    0.003 ffmpeg_reader.py:241(read)
        1    0.003    0.003    0.003    0.003 {method 'read' of '_io.BufferedReader' objects}
        1    0.000    0.000    0.000    0.000 fromnumeric.py:3328(prod)
        1    0.000    0.000    0.000    0.000 fromnumeric.py:69(_wrapreduction)

====================================================================================================

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.003    0.003    0.009    0.009 ffmpeg_reader_cuda.py:207(read)
        1    0.000    0.000    0.004    0.004 ffmpeg_reader.py:241(read)
        1    0.004    0.004    0.004    0.004 {method 'read' of '_io.BufferedReader' objects}
        1    0.000    0.000    0.001    0.001 driver.py:465(function_call)
        3    0.001    0.000    0.001    0.000 driver.py:144(pre_call)
        1    0.000    0.000    0.000    0.000 gpuarray.py:214(__init__)

I noticed that ffmpeg_reader_cuda was also using the ffmpeg_reader for reading the frame which is using numpy to read from buffer:

ffmpegcv/ffmpegcv/ffmpeg_reader.py

Line 251 in f9ce766

img = np.frombuffer(in_bytes, np.uint8).reshape(self.out_numpy_shape)

I was wondering if this could be the problem.

SYSTEM:

• pycuda 2025.1
• ffmpegcv 0.3.16

Thank you in advance!!

[chenxinfeng4]

Thank you for your detailed issue comment. I'd like to explain why the CUDA version might be slower in certain scenarios.

Understanding Performance Differences

The video processing pipeline involves two main compute-intensive steps:

1. Video Decoding: This process converts the raw video stream into an intermediate format.
2. Pixel Transformation: This step transforms the pixel data from sub-sampled pixel space or format to the full (YUV420/NV12 to RGB/BGR).

Python vs FFmpeg Performance

• Python Pipeline:

  • Decoding (ffmpegcv.VideoCapture*)+ Pixel Transformation pix_fmt=bgr24.
  • Pixel Transformation: Transforms the pixel data from YUV to RGB.

• Your FFmpeg Pipeline:

  • ffmpeg -y -vsync 0 -i input.mp4 -f null /dev/null FFmpeg handles only the decoding andskips the additional Pixel Transformation. So it would be faster.

However, when using ffmpegcv.toCUDA(), there are additional overheads due to memory transfers between CPU and GPU, which can further impact performance.

---

Steps to Investigate Performance

To identify why the GPU version is slower, let's break down the processes:

Step 1: Test Decoding Speed

Compare decoding speed using CPU vs. GPU without any pixel transformation.

Mode	Implementation
CPU Decoding	cap = ffmpegcv.VideoCapture(file, pix_fmt='NV12') (High CPU load)
GPU Decoding	cap = ffmpegcv.VideoCaptureNV(file, pix_fmt='nv12') (Low CPU, High GPU)

Step 2: Test Pixel Transformation Speed

Compare pixel transformation speed using CPU vs. CUDA.

Mode	Implementation
CPU Decoding + CPU Trans	cap = ffmpegcv.VideoCapture(file, pix_fmt='bgr24') (Highest CPU load)
CPU Decoding + CUDA Trans	cap = ffmpegcv.toCUDA(ffmpegcv.VideoCapture(file, pix_fmt='nv12')) (High CPU load)
GPU Decoding + CUDA Trans	cap = ffmpegcv.toCUDA(ffmpegcv.VideoCaptureNV(file, pix_fmt='nv12')) (Lowest CPU load. Optimal if memory bandwidth is sufficient)

---

Factors Affecting GPU Performance

Several factors can influence the performance of the GPU version:

• GPU Architecture: The decoding and CUDA performance depend on your NVIDIA GPU's capabilities.
• Memory Bandwidth: Copying frames between CPU and GPU memory consumes time. For example, ffmpegcv.toCUDA(ffmpegcv.VideoCaptureNV(...)) involves two memory copies (GPU → CPU → GPU), which can introduce latency.

---

Conclusion

The slower performance of the CUDA version is often due to the additional overheads of memory transfers and pixel transformations. By isolating these steps, you can better understand where the bottleneck lies.

Let me know your further benchmark. If you have any further questions or need clarification on specific implementations, feel free to ask!

[David-rn]

Hi, thanks for the quick and well explained response!

I still have one more doubt:

For example, ffmpegcv.toCUDA(ffmpegcv.VideoCaptureNV(...)) involves two memory copies (GPU → CPU → GPU), which can introduce latency

Would it be possible to not invove these two memory copies? For example, using cupy as it can read from buffer directly as numpy, but directly on the gpu.

[chenxinfeng4]

This function (np.frombuffer) reads from a buffer without copying data as the buffer is already in CPU memory. It’s efficient and doesn’t add significant overhead. You can test it.

 img = np.frombuffer(in_bytes, np.uint8).reshape(self.out_numpy_shape) 

The two memory copies (GPU → CPU → GPU) can not be easily optimized (GPU → GPU). Because the ffmpeg subprocess can only return the image as the CPU mem, not possible direct (/keep) the image into the GPU mem.

For example, ffmpegcv.toCUDA(ffmpegcv.VideoCaptureNV(...)) involves two memory copies (GPU → CPU → GPU), which can introduce latency

[chenxinfeng4]

The zero memory copy of ffmpegcv.toCUDA(ffmpegcv.VideoCaptureNV(...)) wouldn't be possible until a huge operation on ffmpeg library.

One choice is the VPF publish by NVIDIA, vpf-hardware-accelerated-video-processing-framework-in-python. But the setting up of VPF is a total disaster, I failed.

[David-rn]

Thank you for the explanation, I think I now understand the process correctly.

Is this something related to ffmpeg or Python subprocess?

Because the ffmpeg subprocess can only return the image as the CPU mem.

I'm new using ffmpeg so I'm not sure whether this is correct, but if I understood correctly the option -hwaccel_output_format from ffmpeg would keep the decoded frame in gpu memory?

These questions are just out of curiosity. I guess we can close the issue since the original question was already answered. Again, great library!

[chenxinfeng4]

“Because the ffmpeg subprocess can only return the image as the CPU mem.” ---- yes.. the Python connected ffmpeg by calling the subprocess. The subprocess can only return buffer in CPU mem. “hwaccel_output_format” ---- This option works for ffmpeg, such as transcoding, but the GPU mem data cannot be export directly to Python.

…

---- Replied Message ---- | From | ***@***.***> | | Date | 02/19/2025 21:57 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [chenxinfeng4/ffmpegcv] toCUDA() slower than regular VideoCapture (Issue #64) | Thank you for the explanation, I think I now understand the process correctly. Is this something related to ffmpeg or Python subprocess? Because the ffmpeg subprocess can only return the image as the CPU mem. I'm new using ffmpeg so I'm not sure whether this is correct, but if I understood correctly the option -hwaccel_output_format from ffmpeg would keep the decoded frame in gpu memory? These questions are just out of curiosity. I guess we can close the issue since the original question was already answered. Again, great library! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***> David-rn left a comment (chenxinfeng4/ffmpegcv#64) Thank you for the explanation, I think I now understand the process correctly. Is this something related to ffmpeg or Python subprocess? Because the ffmpeg subprocess can only return the image as the CPU mem. I'm new using ffmpeg so I'm not sure whether this is correct, but if I understood correctly the option -hwaccel_output_format from ffmpeg would keep the decoded frame in gpu memory? These questions are just out of curiosity. I guess we can close the issue since the original question was already answered. Again, great library! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>