[Support]: Memory Leak after upgrading to 0.16 #19646
Replies: 8 comments 7 replies
-
|
Based on your description, you're experiencing memory leaks with ffmpeg processes in Frigate 0.16. The Proxmox logs show that the ffmpeg process is consuming excessive memory (approximately 5.8GB RSS) before being killed by the OOM killer(1). Looking at similar memory issues documented in the community, there have been reports of memory leaks when accessing events, where Python processes consume increasing amounts of memory until the system runs out of resources(2). In those cases, the memory usage increased significantly when viewing events, even without the UI being actively used(2). For your Intel-based system using hardware acceleration, ensure you have the proper configuration. Your current setup uses The documentation shows that for Intel CPUs, you may need to add environment:
LIBVA_DRIVER_NAME: i965Your shared memory allocation of 2048MB should be sufficient based on the calculation guidelines in the installation documentation(4). The tmpfs cache size of 2GB also appears appropriate(4). Since this is a regression from version 0.15.2 to 0.16, and the issue occurs without UI interaction, this suggests a potential memory management issue in the newer version that may need to be reported to the Frigate development team. For immediate troubleshooting, you could try:
📚 Sources:
Have another question? Just tag @inkeep. |
Beta Was this translation helpful? Give feedback.
-
|
@inkeep can you tell me the workaround. Python is taking most of the memory |
Beta Was this translation helpful? Give feedback.
-
|
I have a similar issue. 
-faxxe |
Beta Was this translation helpful? Give feedback.
-
|
You likely need to set a memory limit on the container. There's a known behavior where a python interpreter will use up to 10% of the systems memory. It isn't actually allocated but python doesn't free it. |
Beta Was this translation helpful? Give feedback.
-
|
Also having this problem, frequently noticed it's been down and unreachable, have to go manually reboot the host since it becomes totally unresponsive. As a band-aid I set the system to reboot every hour. |
Beta Was this translation helpful? Give feedback.
-
|
Memory usage can be caused by many different things. At a minimum you should be using TOP to find what process is using memory. It would be much more helpful if you can use memray to analyze exactly what is using memory. |
Beta Was this translation helpful? Give feedback.
-
|
In this case it's ffmpeg not python. If I watch htop inside the frigate container, ffmpeg resident memory usage is increasing at roughly 1MB per second and doesn't stop until the host OOM-kills the ffmpeg process. See attached video of htop. |
Beta Was this translation helpful? Give feedback.
-
|
You may need to adjust your ffmpeg config or make other changes, we haven't seen other reports of steady ffmpeg memory increase like that |
Beta Was this translation helpful? Give feedback.
-
|
@NickM-27 We have seen this too, FFmpeg goes up to 8 GiB (Before OOM) per camera on the recording specific ffmpeg instance. Currently collecting multiple logs, memory allocations, ffmpeg, Prometheus logs, and such and will open an issue soon ; hopefully pinpointing the root cause. |
Beta Was this translation helpful? Give feedback.
-
|
With the help of ChatGPT, I created a log with top of the ffmpeg and phython3 processes. The actual memory consumption RSS is entered every 10 minutes. After only about 2.5 hours of running the script, it is clear that the ffmpeg process is continuously consuming more memory. -faxxe |
Beta Was this translation helpful? Give feedback.
-
|
I'm not really doing anything crazy with ffmpeg, just: @faxxe71 out of curiousity are you also using hwaccel_args: preset-intel-qsv-h264? For now I will try setting a memory limit on the container and see how it goes. |
Beta Was this translation helpful? Give feedback.
-
|
You can try using ffmpeg 5 instead of ffmpeg 7, or adjusting the ffmpeg command. It's not about doing crazy things with ffmpeg, it's about how it interacts with the camera stream. |
Beta Was this translation helpful? Give feedback.
-
@walmartshopper : yes, i use the same hwaccel_args on all my 7 cameras. I will first attempt to set a memory limit and otherwise I will revert to version 15.2. I currently have insufficient time and inclination to delve deeper into this issue. |
Beta Was this translation helpful? Give feedback.
-
|
Based on the thread discussion and your Raspberry Pi configuration, you're experiencing memory issues similar to others using hardware acceleration in Frigate 0.16. Here are the documented workarounds: Raspberry Pi Hardware Acceleration SetupFor your Raspberry Pi configuration using
services:
frigate:
...
devices:
- /dev/video11:/dev/video11You can verify the correct device by running (1): for d in /dev/video*; do
echo -e "---\n$d"
v4l2-ctl --list-formats-ext -d $d
doneMemory Management WorkaroundsBased on the thread discussion, there are several approaches to address the memory leak:
The documentation mentions that a "Fatal Python error: Bus error" is due to a shm-size that is too small, so try updating your shm-size according to the installation guide (2). Alternative Hardware AccelerationIf the memory leak persists with ffmpeg:
hwaccel_args: preset-rpi-64-h265Temporary WorkaroundAs mentioned in the thread, setting container memory limits can serve as a band-aid solution while the underlying memory leak issue is being investigated by the development team. Since this appears to be a regression in version 0.16, you might also consider temporarily reverting to version 0.15.2 until the memory leak is resolved. 📚 Sources: Have another question? Just tag @inkeep. |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone found a solution to this issue yet? Running into the exact same issue about every 7 days with 3 cameras. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Checklist
Describe the problem you are having
After upgrading my frigate docker container to version 0.16, memory usage increases at a rate of about 2GB every 20 minutes until the host runs out of memory and triggers the OOM killer which restarts frigate. This happens without the UI being open.
Steps to reproduce
...
Version
0.16
In which browser(s) are you experiencing the issue with?
No response
Frigate config file
docker-compose file or Docker CLI command
Relevant Frigate log output
Relevant go2rtc log output
Operating system
Proxmox
Install method
Docker Compose
Network connection
Wired
Camera make and model
2x Jidetech 5mp POE Outdoor
Screenshots of the Frigate UI's System metrics pages
proxmox host memory usage taken at the same time as above screenshot:
Any other information that may be helpful
Frigate system metrics page doesn't seem to show the increase in memory, but the proxmox host memory usage graph shows it. The proxmox syslog shows that ffmpeg is the process that was killed by the OOM killer:
Beta Was this translation helpful? Give feedback.
All reactions