[Camera Support]: Camera streams crash all at the same time #10616
Replies: 2 comments 2 replies
-
|
what is your docker compose |
Beta Was this translation helpful? Give feedback.
-
|
The configuration is for a kubernetes cluster. I did notice something yesterday when the crashing happened again. It appears that the container with frigate is restarted just before the issues appear. Which may indicate that either frigate became unresponsive or exceeded the allowed memory and got killed. But I could not figure out the reason for the restart of the container. And I also don't know why a restart of the container would cause all cameras to become unavailable like this. |
Beta Was this translation helpful? Give feedback.
-
|
Ok I just pulled up the server information from my kubernetes cluster and I do see something strange happening right before the frigate restarts. Just before frigate restarts the IOPS of frigate skyrockets: This is mainly triggered by a large amounts of read operations: For some reason this is triggering frigate to restart itself (container seems to be fine as I don't see a new one spinning up). And right after that restart the issue occurs with the streams not working for any of the camera's. |
Beta Was this translation helpful? Give feedback.
-
|
So a new day and a new crash of Frigate. Today all streams crashes again, but at least the cause is identical. What I can figure out from the host machine is the following:
I can also see that normally frigate consumes about 3mb of IOPs throughput a second, but after the restart this drops to 0. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem you are having
Since the upgrade to Frigate 13.x I am experiencing the camera streams all crashing at the same time. At random intervals Frigate reports "No frames received, check error logs" for all cameras at the same time.
Initially I tried to solve the problem by disabling the birdseye view, which did help a bit. In the sense that instead of crashing every few hours it only crashed after a few days.
The logs for Go2rtc show nothing interesting:
I have frigate running in Kubernetes, passing the GPU of the host and a mount to /tmp/cache and /dev/shm to the memory of the host. Frigate is limited to 4Gb of memory and the mounts to the /tmp/cache and /dev/shm are both limited to 2Gb.
Version
0.13.2-6476F8A
Frigate config file
Relevant log output
FFprobe output from your camera
[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"0/0","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":1440,"width":2560},{"avg_frame_rate":"0/0","codec_long_name":"AAC(AdvancedAudioCoding)"}]}},{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"0/0","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":672,"width":896},{"avg_frame_rate":"0/0","codec_long_name":"AAC(AdvancedAudioCoding)"}]}}]Frigate stats
No response
Operating system
Other Linux
Install method
Docker Compose
Coral version
Other
Network connection
Wired
Camera make and model
Reolink and Ezviz
Any other information that may be helpful
No response
Beta Was this translation helpful? Give feedback.
All reactions