Yolov9 model - Coral vs OpenVINO

[PaJaSoft]

Describe the problem you are having

Hey community,
I want to open a discussion regarding Coral models we have now as an option for feeding our object detection demand.

I've been running Coral Mobiledet model (320x320) for long time. Sure, many false positive events generated, but for a real object events happened, the detection was very accurate in my own observation.

Quite recently - approx. 2 months ago I tried another option - https://github.com/dbro/frigate-detector-edgetpu-yolo9/ and comparing this model and its feeded data with other Frigate instance running over Intel CPU+NPU+GPU (Intel(R) Core(TM) Ultra 7 165H).

As of now I can compare - Coral Yolov9 521x512 model (running on dual Coral TPU, interference speed ~20ms) with Yolov9-s 640x640 model (interference speed 15-17ms, 23-28% average NPU usage).

I know Coral Yolov9 model is limited regarding kind of detected objects, but this is not my objection here, we can stick with this limited set of detected object.

It is very obvious to me that Coral Yolov9 model is significantly improved in regards of false positive event, but this 'accuracy' doesn't come from improved detection quality but from the fact many events were simply missed in Coral detection (compared to Intel NPU).

Right now, I replaced and deployed Yolov9 320x320 model instead of its 512x512 variant to compare if my situation can improve (provided accuracy => I would see 100% improvement comparing to Mobiledet, but I can't gather hard data - I don't simply have two Coral devices).

I don't know if the unsatisfied result I was able to gather so far is the result of faulty Yolov9 models from https://github.com/dbro/frigate-detector-edgetpu-yolo9/ or an issue I've incorporated into my own instance (either Coral or Intel NPU), but I can say that my current Coral instance with Yolov9 model running there is pretty the same as my previous Coral Mobiledet deployment.

Performing a benchmark and promote results based on it is nice but as I expect we are focusing on the real cases, not a laboratory environment...

Any idea what I should check/debug/test/whatever on my side either?

Version

0.15 vs 0.16 vs 0.17b1

Frigate config file

Nothing crazy, currently just a detector definition and list of wanted detected object, no any micro adjustment applied (XXX - the same camera/stream).

Note: 15fps for Coral is intentional, just for testing purposes, shouldn't affect (or actually contrary improve) overall results...

Object of interest in both cases:
objects:
  track:
    - bird
    - bicycle
    - car
    - cat
    - dog
    - motorcycle
    - person


Intel CPU+GPU+NPU (Intel(R) Core(TM) Ultra 7 165H):
ffmpeg:
  hwaccel_args: preset-intel-qsv-h265
detectors:
  ov_1:
    type: openvino
    device: NPU
  ov_2:
    type: openvino
    device: GPU
model:
  model_type: yolo-generic
  width: 640
  height: 640
  input_tensor: nchw
  input_dtype: float
  labelmap_path: /labelmap/coco-80.txt
  path: /config/models/yolov9-s-640.onnx        # i:15-17ms, c:23-28%
camera:
  XXX:
    ffmpeg:
      inputs:
        - path: H.265 stream from go2rtc
          roles:
            - detect
    detect:
      width: 2304
      height: 1296
      fps: 5

Coral 2xTPU PCI (AMD Ryzen 9 8945HS w/ Radeon 780M Graphics):
ffmpeg:
  hwaccel_args: preset-vaapi
detectors:
  coral_1:
    type: edgetpu
    device: pci:0
  coral_2:
    type: edgetpu
    device: pci:1
model:
  model_type: yolo-generic
  width: 320
  height: 320
  path: /config/model_cache/yolov9-s-relu6-best_320_int8_edgetpu.tflite
  labelmap_path: /config/labels-coco17.txt
cameras:
  XXX:
    ffmpeg:
      inputs:
        - path: H.265 stream from go2rtc
          roles:
            - detect
    detect:
      width: 2304
      height: 1296
      fps: 15

docker-compose file or Docker CLI command

not important

Relevant Frigate log output

N/A

Install method

Docker CLI

Object Detector

Coral

Screenshots of the Frigate UI's System metrics pages

N/A

Any other information that may be helpful

No response

[hawkeye217]

It's probably best to ask @dbro about this.

[dbro]

Hello, the responses already provided are excellent and I agree that it's worth exploring if different threshold parameters will improve the rates of false positives and false negatives for you.

Keep in mind that the training data for both of these freely available Coral models (Mobiledet and YOLO v9) is COCO, which is set of images that include many examples that are different from what would be seen in a surveillance camera feed. Check out the example images from COCO here to see how different they are. It would be better to use images compiled from Frigate users -- this is exactly the kind of images that are used to train Frigate+ models that are available to paying subscribers. With that caveat in mind, there is some accuracy performance data here. But this evaluation also uses COCO data, so may not be representative of the images that surveillance cameras see.

[NickM-27]

To be clear, the coral model is quantized, so there will be an accuracy loss compared to the onnx model that runs on Intel. That will always be the case.

The initial version of that model had issues which made it unusable in my experience, but the user worked diligently and improved the performance of the model through various techniques, so it is much improved compared to that point. If you have false negatives you can always look at lowering your threshold and min scores, the default values are optimized for the default coral model.

[PaJaSoft]

Easiest way is to save a recording clip and then rerun it through frigate with the debug live view.

Definitely will try that - the accurate time I can get from OpenVINO instance and just let it run again against Coral TPU would be easy... however I don't know how to do it.

Let say I know what is an interesting part in the past however not recorded as an event on that Frigate - I know how to re-play this part again from history, but don't know how to debug and see what Frigate things about this sequence now.

[hawkeye217]

but don't know how to debug and see what Frigate things about this sequence now.

See https://deploy-preview-19787--frigate-docs.netlify.app/troubleshooting/dummy-camera

[PaJaSoft]

but don't know how to debug and see what Frigate things about this sequence now.

See https://deploy-preview-19787--frigate-docs.netlify.app/troubleshooting/dummy-camera

YEA! That's exactly I was looking for and missed it in the new draft of the doc. Thank you so much for pointing me there!

[PaJaSoft]

I'm thinking about the published limitations of dummy camera debug. Is there a way how to track what is happening between Frigate and Detector - simply debug/trace that into the text log? I can inject this into the Python code for sure, but asking if something is already baked there for similar debug purposes and can be just enabled...

[NickM-27]

In most cases there's no reason to go that deep.

[ristoavila]

I'm also buzzled with this. I changed with the 0.17 to yolov9 Coral usb but it seems that it now fails to identify even simpler cases of person and cars that were detected with the Mobiledet. I'm thinking should I reduse the filter threshold and min_score values to compensate less false positives to still capture the cases.
As an example I have PIR sensors which alerts when something large enough actually moves so there was car driving by and yolov9 did not detect it even though it was visible (yes a bit blurry but clearly a car). Another one was two kids running across the camera feed and there was stand alone bike only the bike was detected. Need to do maybe some more debugging of the models.

[taltosgr]

I am very happy with a single Coral PCIe using Frigate+. I now have almost no false positives and generally it seems to be much better detecting during both night and day (I use four Reolink CX410 cameras that have night color vision, so no infrared training was ever needed).
I have uploaded about 5000 images to Frigate+, so even using the mobiledet Frigate+ model the recognition was decent, but now it is so much better with almost no false positives. It also doesn't seem to be missing anything I have it configured for.
Coral support for YOLOv9 using Frigate+ models seems to have extended the usage lifetime for the little device (at least for me). Lately I had been considering it, but for now I see no reason to migrate to e.g. OpenVINO for better accuracy.
Overall I am very happy with the results and it seems to be even more responsive and to follow objects better than before with little inference loss (previously it was around 7-8 now it is around 10-13). Even so I have almost no skipped detection frames, even on windy days with much motion.

[ristoavila]

how do you know if you missed anything? I'm not too worried about the false positives but the fact that it missed cars and persons, with out PIR sensors I would not have noticed this at all. Of course I'm not using yet Frigate+ models but I did not use it for Mobiledet either.

[taltosgr]

I started with mobiledet on OpenVINO and after that had also tried out a YoloNAS model that was somewhat more accurate for my OpenVINO configuration before I got the Coral, but I wanted to achieve detection with a lower power consumption (and noise, since the CPU cooler of my 1L PC would ramp up with YoloNAS/OpenVINO).
I record motion 24/7, and usually go through the motion detection timeline in the evening to check if anything stands out, therefore I said that the accuracy is very good. Actually, for me the issue never was about not detecting something, but more about false positives, since it used to detect my dogs or cat as a person and I would be getting a lot of frigate alert HA notifications that were false alarms. Now I get a false alarm maybe once every month, which is quite an improvement compared to almost daily!