[Support]: Memory Leaking in Docker/Proxmox #21222
Replies: 6 comments 16 replies
-
|
I can see you're experiencing significant memory growth in your Frigate Docker container. Based on your configuration and the documentation, here are several areas to address: Configuration IssuesYour For Intel hardware acceleration, you're using Restream ConfigurationYour cameras are using go2rtc restreams, but you have some inconsistencies. For the When using go2rtc restreams, you should reference them as: inputs:
- path: rtsp://127.0.0.1:8554/front_door
input_args: preset-rtsp-restream
roles:
- detect
- recordMemory and Recording IssuesYour logs show: "No new recording segments were created for front_door in the last 120s."(1) This suggests potential stream stability issues that could contribute to memory problems. The Recommendations
Without access to more detailed system metrics during the memory growth period, it's difficult to pinpoint the exact cause, but these configuration adjustments should help stabilize your system. 📚 Sources: Have another question? Just tag @inkeep. |
Beta Was this translation helpful? Give feedback.
-
|
You'll need to use |
Beta Was this translation helpful? Give feedback.
-
|
By calling the frigate restart api |
Beta Was this translation helpful? Give feedback.
-
|
Can I just restart Frigate through the UI? |
Beta Was this translation helpful? Give feedback.
-
|
Yes |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the info, really appreciate it. Might be a bug, restarting Frigate doesn't automatically generate the HTML reports. Will investigate the manual reports to see where the memory leak is |
Beta Was this translation helpful? Give feedback.
-
|
I tried looking through the documents but didn't notice anything unusual that's contributing to the memory leak. However I'm not well versed in analyzing these reports. Any ideas? |
Beta Was this translation helpful? Give feedback.
-
|
Adding one more to this as I'm having the same issue. Hypervisor: Proxmox 9.1.2 The host machine has an Intel N150 and I have the iGPU passed through. My instance has ~2gb/hr continuous RAM consumption with 3 cameras. It eventually uses all available ram, gets OOM killed, and repeats. It's the exact same symptoms everyone else has. One thing I've noticed though is if I disable recording the memory leak does not happen. The leak only occurs with recording enabled. I have tried capturing everything via memray and processing all reports. None of the reports show the increasing memory usage (so I haven't bothered to attach them - they show nothing useful). To be specific, the list I tested was:
Memray as it's configured isn't revealing the memory leak. For anyone else having the same issue - if you globally disable recording does the leak stop? |
Beta Was this translation helpful? Give feedback.
-
|
If memray doesn't show anything then it's not python that's using the memory, so you need to use other tools to find where it is. It might be in /tmp/cache |
Beta Was this translation helpful? Give feedback.
-
|
Have you disabled ballooning? https://deploy-preview-19787--frigate-docs.netlify.app/frigate/installation#proxmox |
Beta Was this translation helpful? Give feedback.
-
|
I have the LXC Swap set to 0, and still getting memory leaks. Also, @jasonlaguidice, I too don't get the leak once I disable record. |
Beta Was this translation helpful? Give feedback.
-
Are you sure you aren't recording to ram directly? You might miss the mountpoint somewhere and data lands in tmpfs instead. |
Beta Was this translation helpful? Give feedback.
-
|
Apologies about the late reply to all. I AM using a temp file system in ram, defined as such in the stack: But if I'm not mistaken that shouldn't result in infinitely ballooning RAM usage since there is a 1gb cap. For Hawkeye - ballooning is not an option for an LXC. That's only applicable to running a VM for the purpose. So from this - seems like nothing is directly wrong with my setup but the RAM usage runs away if record is enabled. |
Beta Was this translation helpful? Give feedback.
-
|
for me the problem got solved by running frigate as an OCI instead of an LXC with docker. |
Beta Was this translation helpful? Give feedback.
-
|
I'm experiencing the same exact issue. Memory leak with 0.16.x (not fixed in 0.16.4 either). I've back-revved to 0.15.2 which does not exhibit the memory leak. What's the hope to get this fixed? |
Beta Was this translation helpful? Give feedback.
-
|
Me too... proxmox and memory leaks. Sometimes (rarely) restarting proxmox lxc-> debian is not enough, and i have to restart the proxmox machine because all other lxc gets swamped. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem you are having
I'm seeing memory runaway at a rate of 2GB/Hr on my Frigate install running in Docker in a Debian 12 LXC. I've scoured documentation, used the Frigate AI chat, checked prior issues, tried changing to rtsp-generic, installed on docker bare-metal on my proxmox host (to rule out sysinfo conflict being the culprit), and still seeing the same thing. I set the docker memory limit and swap limit to 4gb to prevent it from bogging down the entire system. Frigate logs don't show any errors while this is happening until I hit the 4gb memory limit and errors start.
Frigate container memory (via Portainer)
Version
0.16.3-9034454
What browser(s) are you using?
No response
Frigate config file
Relevant Frigate log output
Relevant go2rtc log output
FFprobe output from your camera
Don't have that command inside my containerFrigate stats
No response
Install method
Proxmox via Docker
docker-compose file or Docker CLI command
Object Detector
OpenVino
Network connection
Mixed
Camera make and model
driveway= Amcrest ASH22-W ; hallway = Amcrest ASH21-B-V2 ; front_door = Reolink E1 Pro, nanit = restream from indiefan/nanit github
Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpful
Proxmox server is wired, but all cameras are wifi.
Beta Was this translation helpful? Give feedback.
All reactions