CPU usage goes up when hwaccel is enabled on Raspberry Pi 4?

[DracoC77]

Curious if anyone else has seen their CPU usage actually go up when they enable HW Accel on a 64-bit OS Docker instance via Raspberry Pi 4? (h264_v4l2m2m).

This has been a problem ever since I first set up frigate, was hoping the new 0.11.0-rc1 release would magically fix it but alas it's having the same issues.

GPU memory is set to 256 and shm is set to 600mb which should be more than enough for the 3x cameras I'm running. Confirmed my docker image is running in priveleged mode, also mapped all of the /dev/video* into the docker container in case that was necessary.

Tried applying hw_accel globally and also only applying the hw_accel args to the input stream/decode of specifically a few 1080p cameras (since the official raspberry pi documentation suggests thats the limit of decoding hw accel support)

Any other suggestions/things to try would be appreciated!

Here's a plot of CPU usage without any hw_accel arguements and after I turn it on.

[NickM-27]

Frigate 0.11 has a new ffmpeg build so if it is still occurring then seems to be expected or a deeper issue. That graph isn't super helpful, a screenshot of htop run inside the frigate container before and after would be able to see how things are being affected.

What resolution are your cameras? It is entirely possible that it is less work to decode the stream for the CPU (when you factor in the CPU usage needed to move the stream to / from GPU) if the cameras are low enough resolution

[ediguidedog]

I noticed something similar with one of the earlier betas. Depending on which qsv arguement I used the CPU usage varied enormously. The old style args were less CPU intensive than the new but that's now reversed since beta 7. More info in #3482 .

I'm now using vaapi over qsv as for some reason that lowered it even further. There is a discussion somewhere about that but I can't find it ATM.

[ikornaselur]

Just to add to this as I'm testing Frigate out right now. Using a Raspberry Pi 4B 4GB with two cameras to test with. Without hwaccel the CPU usage sits at around 35-45%, but if I enable it it shoots up to 85-90%.

I'm deploying frigate with the helm chart on a k3s cluster btw, but specifying the tag to be 0.11.0-rc1

The following shows the CPU graph from Netdata running on the node, right around the middle I restart the process to turn hwaccel of:

I'm just trying to set

  hwaccel_args:
    - -c:v
    - h264_v4l2m2m

directly on the cameras, or globally, running 0.11.0-rc1 at the moment. I've increased the gpu ram to 512MB as well.

Happy to debug further for more info.

[ikornaselur]

Ah, I thought I had mentioned it in my post! Sorry, it's 1x1080p and 1x640p, so not exactly a lot. I guess it's similar to what @xconverge is seeing, that it's just more efficient on the CPU since it's not that intensive already.

I was hoping though to add 2-3 more cameras in a bit, which makes me wonder if the Raspberry Pi 4B just isn't going to handle it? I thought that with hwaccel it should be fine, as I have a Coral TPU for the recognition, but if it's already struggling with the ones I have, I may need to rethink my plans.

[NickM-27]

Make sure your detect resolution in the config exactly matches the true resolution of the camera so it doesn't have to resize

[xconverge]

That made a huge difference for me @NickM-27 ! I was using 720p for my detect, so it was resizing very often, both with hw accel and no hw accel, CPU usage is much better now! Had to tweak my zones and masks but it was worth it, awesome

[NickM-27]

That made a huge difference for me @NickM-27 ! I was using 720p for my detect, so it was resizing very often, both with hw accel and no hw accel, CPU usage is much better now!

Yeah, resize isn't done in hardware so that can use quite a bit of CPU

[xconverge]

I am now down to around 10% usage with hwaccel enabled and very happy! interesting

[xconverge]

Yes I am seeing the same thing and have been scratching my head over it for a few days

1 stream at 1080p

without hwaccel, ~25% cpu

with hwaccel (h264_v4l2m2m) ~75% cpu

I dont have a benchmark to compare it against in past releases because ffmpeg was crashing after a few minutes with any of the hwaccel settings, now it runs longer just with a very high cpu usage

[NickM-27]

That might make sense, it does require CPU to move the frame between the GPU memory and general memory as well as other factors, and the CPU is probably more capable of 1080p relative to that work.

A 4K camera stream would likely be a different story

[xconverge]

I will stop scratching my head then, I agree that it seems possible, just...a pretty big jump!

[NickM-27]

Yeah, I've seen behavior with my ryzen 5700g server. Watching htop running on one of my 5 MP stream it goes 50% to 25% CPU time. Meanwhile my 1080p camera is the same as without.

[DracoC77]

This sounds like our answer then, qualitatively it seems like 1080p cameras are ever so slightly more efficient on just CPU than trying to use hwaccel while larger cameras fare better on hwaccel. I also have a 5MP stream I haven't been playing with but I might do some testing later on that on my RPi4 just to confirm nothing else weird is going on.

[NickM-27]

@DracoC77 had a note on a different thread but it is important to make sure you are setting the detect -> height / width exactly as the resolution is for the detect stream, otherwise frigate uses CPU to resize the stream. Another user wasn't doing this and saw greatly improved results

[DracoC77]

I did some more testing based on various comments made in this thread:

General Troubleshooting

• Confirmed that my detect resolution matched the stream resolution with ffprobe (and we weren't trying to do any additional rescaling due to misconfiguration there)
• Confirmed shm size was much larger than the problem at hand (600mb vs. one 2560x1920 or two 1920x1080 cameras)
• Increased gpu_mem on RPi4 to 512MB to make sure its not running out of memory

Tested 2x 1920x1080 camera streams W/O hwaccel and WITH hwaccel:

• Resultant CPU usage seemed somewhat like a wash (higher variation but means looks to be about the same?)
• Does this imply that for 1080p, using hwaccel on a RPi4 makes no difference? If so what's the point of hw acceleration if it doesn't reduce CPU utilization?
  

Tested 1x 2560x1920 camera stream W/O hwaccel and WITH hwaccel:

• Per @NickM-27's comments, was expecting to see an improvement in CPU usage but I was unable to get the WITH hwaccel configuration working
• W/O hwaccel the stream worked completely, WITH hwaccel I was only seeing a green screen and occasional CPU bursts that were fully system utilization
• Tried increasing GPU memory from 256 to 512 in case that was limiting factor, no dice
• Errror I was getting from frigate:

frigate | [2022-08-04 21:29:55] frigate.video ERROR : driveway_test: ffmpeg process is not running. exiting capture thread...
frigate | [2022-08-04 21:30:05] watchdog.driveway_test ERROR : Ffmpeg process crashed unexpectedly for driveway_test.
frigate | [2022-08-04 21:30:05] watchdog.driveway_test ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
frigate | [2022-08-04 21:30:05] ffmpeg.driveway_test.detect ERROR : [segment @ 0x55a5017440] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
frigate | [2022-08-04 21:30:05] ffmpeg.driveway_test.detect ERROR : Finishing stream 1:0 without any data written to it.

• Running on a Reolink E1 Zoom with the following config files:
  frigate_config.txt

[NickM-27]

Interesting, I run my reolink with the http://rtmp and VA API (obviously not on an RPi). I am surprised that it doesn't like the hwaccel but without works okay.

[DracoC77]

Yea I was surprised it wasn't working either (I tried both http:// direct and also rtsp:// feeds and it behaved the same way), I suspect that it's because the RPi4 doesn't support hw decoding of H.264 above 1080p60 per their specs:

H.265 (4Kp60 decode);
H.264 (1080p60 decode, 1080p30 encode);
OpenGL ES, 3.0 graphics

Also I'm still suspicious that with and without hwaccel on 1080p decode is a wash, it makes me think either hwaccel isn't working properly, there's something wrong with the docker/container build, or maybe its not as beneficial at lower FPS we use for detect vs. say 1080p60fps? (but if we're saying it takes the same CPU to move the 1080p frames to memory and hwaccel decode as it takes for the CPU to just decode the frames, then that statement should be true at any FPS no?)

If folks have any other thoughts or testing on RPi around hwaccel, would be much appreciated. I wonder if its worth submitting an issue for further investigation.

[xconverge]

I suspect it's just not performant and ready yet still

Stuff like this:
jc-kynesim/rpi-ffmpeg#42

[NickM-27]

Also I'm still suspicious that with and without hwaccel on 1080p decode is a wash,

To be clear I wasn't saying it's always the case, just that it can less effective be in certain circumstances

detect vs. say 1080p60fps? (but if we're saying it takes the same CPU to move the 1080p frames to memory and hwaccel decode as it takes for the CPU to just decode the frames, then that statement should be true at any FPS no?)

It doesn't work like that. The ignoring of fps happens after the hardware acceleration so even if you set detect fps in frigate as 5 all whatever fps are still decoded.

It also depends on the camera bitrate, encoding profile (H264 Baseline, Normal, High, etc)

Comparing 2 of my 5 MP cameras for example they have different CPU / GPU % usage because they are different bitrate

If folks have any other thoughts or testing on RPi around hwaccel, would be much appreciated. I wonder if its worth submitting an issue for further investigation.

We've had a number of rpi users testing this. Before 0.11 moved rpi to jellyfin it was entirely broken and jellyfin fixed it for those users and provided much improved results in their case