Detection is inconsistent when loop testing

[forkineye]

I've been fine tuning the detection for one of my cameras and am running into issues with repeatability / consistency. I took a clip from my 24/7 recording that didn't detect last night (same stream is used for detection), and set it up as a test camera. The mask, zones, motion, etc... are all setup just as they are on the physical camera (listed below). The issue that I'm having is that this 60sec clip is not consistently being detected. Sometimes, it'll detect every minute or two, sometimes it will take 5-15min. The stream from the camera (Reolink RLC-810A) is 20fps rtmp ( it's the ext sub-stream that can't be configured) and configured for 10fps in Frigate. I've tested with it configured for 20fps as well and still experience issues. Example of what I'm seeing and setup / camera / stats data below. Any ideas as to why this is happening? I'm thinking it may have something to do with how Frigate chunks everything up into 60sec clips? Does this happen for detection as well?

Setup:

Dell T420 - 2x E5-2450L - 64GB RAM
Proxmox 7.0
Ubuntu 20.04 VM (32GB RAM, 8 CPUs)
Frigate 0.8.4-5043040 in Docker
Coral Mini PCIe

Camera config:

cameras:
  test:
    ffmpeg:
      inputs:
        - path: /media/frigate/ndtest1.mp4
          input_args: -re -stream_loop -1 -fflags +genpts
          roles:
            - detect
    width: 896
    height: 512
    fps: 10
    motion:
      contour_area: 40
      frame_height: 180
      mask:
        - 896,0,896,92,147,88,141,201,0,234,0,81,0,0
        - 896,270,896,209,850,191,824,220,849,266
    objects:
      track:
        - person
        - car
      filters:
        person:
          min_score: 0.4
          max_area: 8000
        car:
          mask:
            - 0,512,896,512,896,0,738,0,734,77,782,137,764,181,654,221,348,222,247,218,178,186,164,147,254,80,251,0,0,0
    zones:
      front_yard:
        coordinates: 896,512,896,375,720,237,493,254,297,252,232,245,150,230,0,284,0,512
        filters:
          person:
            threshold: 0.5
      front_alarm:
        coordinates: 896,342,753,269,195,282,0,512,896,512
        filters:
          person:
            threshold: 0.5
    snapshots:
      enabled: True
      bounding_box: True
      retain:
        default: 1

Stats:

{
  "detection_fps": 7.0, 
  "detectors": {
    "coral_pci": {
      "detection_start": 0.0, 
      "inference_speed": 9.86, 
      "pid": 43
    }
  }, 
  "driveway": {
    "camera_fps": 5.1, 
    "capture_pid": 58, 
    "detection_fps": 0.0, 
    "pid": 50, 
    "process_fps": 5.1, 
    "skipped_fps": 0.0
  }, 
  "front_porch": {
    "camera_fps": 5.1, 
    "capture_pid": 62, 
    "detection_fps": 0.0, 
    "pid": 51, 
    "process_fps": 5.1, 
    "skipped_fps": 0.0
  }, 
  "living_room": {
    "camera_fps": 5.1, 
    "capture_pid": 68, 
    "detection_fps": 0.0, 
    "pid": 53, 
    "process_fps": 0.0, 
    "skipped_fps": 0.0
  }, 
  "service": {
    "storage": {
      "/dev/shm": {
        "free": 125.7, 
        "mount_type": "tmpfs", 
        "total": 134.2, 
        "used": 8.5
      }, 
      "/media/frigate/clips": {
        "free": 1057442.4, 
        "mount_type": "zfs", 
        "total": 1930527.2, 
        "used": 873084.9
      }, 
      "/media/frigate/recordings": {
        "free": 1057442.4, 
        "mount_type": "zfs", 
        "total": 1930527.2, 
        "used": 873084.9
      }, 
      "/tmp/cache": {
        "free": 693.4, 
        "mount_type": "tmpfs", 
        "total": 1000.0, 
        "used": 306.6
      }
    }, 
    "uptime": 2970, 
    "version": "0.8.4-5043040"
  }, 
  "street": {
    "camera_fps": 5.1, 
    "capture_pid": 55, 
    "detection_fps": 0.0, 
    "pid": 48, 
    "process_fps": 5.1, 
    "skipped_fps": 0.0
  }, 
  "test": {
    "camera_fps": 10.0, 
    "capture_pid": 54, 
    "detection_fps": 7.0, 
    "pid": 46, 
    "process_fps": 10.0, 
    "skipped_fps": 0.0
  }
}

[blakeblackshear]

If the resolution/fps of the test camera file matches, I would expect it to perform similarly. Not everything is exactly the same, but it should be about as reliable. I would suggest watching the test camera in debug view. Do you see motion boxes? Do you see regions around the motion boxes? Are bounding boxes detected in that region?

[forkineye]

I'm seeing motion, region and bounding boxes. The recording is here if you would like to analyze it. Is there any debugging I can enable to help figure out what's going on (evaluation / weight data, etc...)?

[tve]

Could the fact that it's a loop throw it off a bit, or introduce enough variation that it doesn't detect consistently? Maybe you could freeze the first frame for some 10 seconds at the start of the loop to try and give it a dead moment?

[forkineye]

Whenever it does detect, it's always right at the end where the video stops. I have the following clip too, I'll merge them together and test again. I'm curious though if it has something to do with the segmentation and these events falling on the edge of those segments? For instance, this is was the config looks like:

  "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -use_wallclock_as_timestamps 1 -i rtmp://192.168.2.102/bcs/channel0_ext.bcs?channel=0&stream=0&user=XXX&password=XXX -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /media/frigate/recordings/street-%Y%m%d%H%M%S.mp4 -c copy -f flv rtmp://127.0.0.1/live/street -r 10 -f rawvideo -pix_fmt yuv420p pipe:", 
          "roles": [
            "detect", 
            "rtmp", 
            "record"
          ]
        }, 

[forkineye]

Even with the concatenated two minute clip, the same issue occurs. It seems to reliably detect the first time around, then is sporadic thereafter (usually failing to trigger a snapshot). The concatenated clip is here.

[lawm]

Default threshold is .7 and your Events screenshot shows the score, 70.70%, is very close to it.

Lowering the threshold should get better detection.

I don't know why it would be sporadic though.

[blackgold9]

Id love to see the ability to set up a motion/AI test based on a segment of recorded video included in the UI.