Frigate Tip: Best Practices for Training Face and Custom Classification Models

[hawkeye217]

The Frigate Tips series is a collection of posts where the Frigate developers give tips on how to best configure, use, and maintain your Frigate installation. Frigate 0.16 introduced Face Recognition and Frigate 0.17 beta introduces Custom Classification for state and object classification. This post’s tip focuses on how to train these models effectively and, just as importantly, what not to do.

Training AI models: more images is not always better

A common instinct when training any AI model is to select as many images as possible, especially when the UI presents a large batch of examples. While this feels intuitive, it often leads to worse results for both face recognition and state/object classification in Frigate.

The key idea to understand is this:

Diversity matters far more than volume.

Selecting dozens of nearly identical images at once is one of the fastest ways to degrade model performance. This is why Frigate does not implement a way to bulk train faces or classification images.

---

Face recognition: avoid bulk batches of similar images

Frigate’s face recognition uses an ArcFace-based model, which learns to represent faces as embeddings in a feature space. These models are very good at learning discriminative features, but they can also overfit when trained on narrow, repetitive data.

Why bulk training hurts face recognition

If you select a large batch of highly similar images—such as sequential frames from the same clip with the same lighting, pose, and expression—the model has very little incentive to learn robust facial features. Instead, it learns to rely on incidental details that only exist in that capture scenario.

In practice, this means:

• The model may score extremely high on the images you trained
• Recognition quality drops when:
  • Lighting changes
  • The face is partially occluded
  • The angle or expression differs
  • The camera distance changes

This is classic overfitting behavior and is well-documented for ArcFace-style embedding models.

Best practices for Face Recognition training

The biggest thing to keep in mind is that training is a process, not a one-time event.

• Start small: Start with 1-5 high quality, front-facing shots of each person you want to detect. These don't necessarily need to be from your security cameras - in fact, one or two photos from a quality mobile phone camera is an excellent starting point.
• Iterate slowly: Once you have uploaded and trained those, start training more from the Recent Recognitions tab in the Face Library as they are detected by your cameras. Prefer front-facing images and larger area images first, and ignore ones that see faces from an angle. Aim for perhaps 20 total images in this stage.
• Prefer variation over quantity: Then move on to training images with:
  • Different lighting (avoid infrared/grayscale images)
  • Slightly different angles
  • Different expressions
  • Try to avoid training images that are scoring above 90% in this stage
• Only add more when necessary: Once you have 30 or more images for each person, then start training ones from more difficult angles. Add new images only when recognition struggles or produces low-confidence matches.
• Avoid bulk selection of many nearly identical frames, especially from the same moment in time.

---

State & object classification: the same principle applies

State and object classification models in Frigate use MobileNetV2, a lightweight convolutional neural network designed for fast CPU inference. While this model is very different from ArcFace, it shares a critical property with all CNN-based classifiers:

They can overfit quickly when trained on homogeneous data.

Why bulk training hurts state and object classification

State classification models are trained on fixed camera crops, and object classification models are trained on tracked object crops. When you select many images that all look nearly identical—same lighting, same shadows, same background noise—you are not teaching the model what defines a state or class. You are teaching it what that exact moment looked like.

This often leads to models that:

• Work perfectly under the original conditions
• Become unstable or flip states when:
  • Day turns into night
  • Weather changes
  • Seasonal lighting shifts
  • Minor scene noise appears (insects, reflections, temporary objects)

Because MobileNetV2 learns visual patterns directly from pixel data, lack of intra-class variation causes the model to rely on fragile cues that don’t generalize well.

Best practices for state & object classification

• You do not need to train every example or all classes shown in the wizard: You can simply continue if you are unsure about a state or an object and add more as they are added to the Recent Classifications tab.
• Start with a small, representative set per class: Remember, diversity means much more than volume.
• Favor “hard” examples: As Frigate begins adding images to the Recent Classifications tab, favor:
  • Images scoring below 90-100%
  • New lighting or weather conditions
  • Unusual but valid appearances
• Avoid training large batches of images that already score 100% (or close to 100%)
  • These add little new information
  • They increase the risk of overfitting

In most cases, models begin working well with surprisingly few examples and improve naturally over time.

---

The wizard is just the starting point

The classification wizard is designed to get a model up and running quickly using readily available data. It is not meant to force you to find and label every possible state or class upfront.

Going back through historical recordings to locate rare conditions would add significant UI and workflow complexity for limited benefit, which is why we've chosen not to implement it.

Instead, Frigate is designed to work iteratively:

• Missing states or classes will naturally appear in Recent Classifications
• Those images tend to be more valuable because they represent:
  • New conditions
  • Edge cases
  • Scenarios the model hasn’t seen before
• Training incrementally from this view produces more stable and reliable models over time

---

FAQ

Why does my model score 100% but still behave incorrectly?

A 100% score only means the model is very confident based on what it has already learned. If the training data lacked diversity, the model may be confidently wrong under new conditions. This is a common sign of overfitting. Delete some images from your classes and start again with the suggestions above.

Why does my state classification flip between states at night or during weather changes?

This usually means the model has not been trained on those conditions. Nighttime, rain, snow, or seasonal lighting changes often introduce visual patterns the model has never seen. Adding a small number of representative examples from those conditions usually stabilizes the model.

Why don’t I see all states or classes during the wizard?

The wizard only shows a sampling of images that exist in your recordings. Some states or classes may not have occurred yet. This is expected. Don't worry about finding images for every class as you progress through the wizard. If something is obvious, then select it. If you're unsure, you can leave it unselected.

Why don’t I see updates coming into the Recent Classifications tab? I've set up an interval in my config, shouldn't I be seeing them every n seconds?

Images are not saved when the state is stable (detected state matches current state) and the score is 100%. This prevents unnecessary storage of redundant high-confidence classifications. There are a number of other conditions as well, see the official docs.

How do I train missing states or classes later?

Use the Recent Classifications view. When a new state or class appears, label it there and retrain the model. These images are often more valuable than those shown during the initial wizard.

---

Official Frigate documentation

Find more tips, suggestions, and configuration options in the official documentation.

• Face Recognition: https://docs.frigate.video/configuration/face_recognition
• State Classification: https://docs-dev.frigate.video/configuration/custom_classification/state_classification
• Object Classification: https://docs-dev.frigate.video/configuration/custom_classification/object_classification

---

If this has been a helpful post, give it an upvote and thumbs up. If you have any questions, leave them down below for us!

— The Frigate dev team (Blake @blakeblackshear, Nick @NickM-27, Josh @hawkeye217)

[yngveskog]

Thanks for very good tips, but as a man of course I was reading the manual after implementing this feature, so now I know that I have done everything wrong... From the documents (https://docs.frigate.video/configuration/face_recognition/) I see that it's possible to start over again, but I'm missing the /media/frigate/clips/faces directory. The closest I have is /media/frigate/clips/person. I can't find any other options to delete all the faces

[hawkeye217]

If you're missing that directory, it would indicate that your Docker mount points are not set up correctly.

[willliamchan]

Thank you again for the great work! I am currently exploring the possibility to replace doubletake+rekognition with Frigate Face Recognition, after few weeks of testing plus reading the documentation and the tips here I have a few questions.

• Start with color front facing picture as a start which make total sense, is that means afterward I can slowly add picture to the model that not actually looking at the camera or it will start mess up the detection?

• Follow up question re greyscale faces, there are some people who only visit my place at night and there is no way for me to get a face during day time, is it better not include those people in the training set or I can still start with greyscale images?

• I have a face set that contains color and greyscale faces, by deleting just the faces in greyscale will it trigger the re-train with only existing pictures or need to start everything over again?

• As mentioned I have been testing it for a few weeks of time (about 15 - 20 pics per face), and until this point it still detect person as someone else, what is the suggestion to correct/improve it?

• The system keeps detecting a delivery person as myself, while I do not want to create a face for the delivery person (as I don't know his name). What should I do to avoid that from happening?

• Lastly, what is the best practice when it detected as wrong person? Let say I have 20 pics trained for Tom, and 20 pics trained for Sam. And when the system detected Tom as Sam, I click the pic and assign it back to Tom and let the model re-train. And when I refresh the pic after a few seconds it stays the pic is Sam.

Million thanks!

[hawkeye217]

Thank you again for the great work! I am currently exploring the possibility to replace doubletake+rekognition with Frigate Face Recognition, after few weeks of testing plus reading the documentation and the tips here I have a few questions.

* Start with color front facing picture as a start which make total sense, is that means afterward I can slowly add picture to the model that not actually looking at the camera or it will start mess up the detection?

Yes — that’s the correct approach. Start with clear, front-facing, well-lit color images first. Once recognition is stable, you can gradually add variation (slight angles, expressions, lighting changes).

What you want to avoid early on is adding:

• strong side profiles
• poor lighting
• motion blur

Adding those too early makes the embedding less stable and increases false matches.

* Follow up question re greyscale faces, there are some people who only visit my place at night and there is no way for me to get a face during day time, is it better not include those people in the training set or I can still start with greyscale images?

ArcFace models (including those commonly used with MobileNetV2) are trained on RGB face datasets (e.g. MS-Celeb-1M, VGGFace2). They are not explicitly trained on grayscale or IR imagery.

That means:

• Grayscale images do work technically (they’re usually replicated across RGB channels),
• But they live in a different feature distribution than daytime color images,
• Which increases the chance of embedding overlap and misclassification.

This is why most ArcFace guidance recommends:

• Training primarily with color images
• Avoiding grayscale unless necessary

Best practice:

• Start with daytime color images only
• Add grayscale images later, once recognition is stable
• Keep grayscale samples very limited and reasonably clear

If someone only appears at night, grayscale training is fine — just avoid mixing low-quality IR images with good daytime ones early on.

* I have a face set that contains color and greyscale faces, by deleting just the faces in greyscale will it trigger the re-train with only existing pictures or need to start everything over again?

Yes — deleting those images is enough. Frigate retrains using whatever images currently exist for that person. You do not need to start over or delete the identity entirely.

* As mentioned I have been testing it for a few weeks of time (about 15 - 20 pics per face), and until this point it still detect person as someone else, what is the suggestion to correct/improve it?

This is usually a quality issue, not quantity. Common causes:

• Too much similarity between people (hair, glasses, facial hair, etc.)
• Low-quality or blurry images mixed in
• Too little variation in pose or lighting

Suggestions:

• Remove the worst 20–30% of images
• Aim for ~15–30 high-quality samples
• Add diversity gradually

* The system keeps detecting a delivery person as myself, while I do not want to create a face for the delivery person (as I don't know his name). What should I do to avoid that from happening?

This is expected behavior and should not be “trained away.”

Do not:

• Create a face profile for them
• Reassign their image to yourself

That pollutes your embedding and makes recognition worse.

Instead:

• Leave it unassigned / unknown
• Or ignore the detection

Face recognition learns who you are, not who you are not.

* Lastly, what is the best practice when it detected as wrong person? Let say I have 20 pics trained for Tom, and 20 pics trained for Sam. And when the system detected Tom as Sam, I click the pic and assign it back to Tom and let the model re-train. And when I refresh the pic after a few seconds it stays the pic is Sam.

Reassigning a detection does update the model, but the effect is weighted against all other samples for that person.

So if Tom has 20 images and you correct 1 misclassified frame, that single sample won’t shift the embedding much. The model still reflects the overall dataset, which is expected behavior for ArcFace-style embeddings.

If misclassification continues:

• Remove low-quality or misleading images
• Add a few high-quality samples to the correct person
• Avoid repeatedly reassigning the same frame

[willliamchan]

Thank you very much!

[ds-sebastian]

I tried to create this tool to pull diverse images from Immich using their defined faces (which I believe uses clustering) and then selects the most diverse subset based on Farthest Point Sampling of the image embeddings:

https://github.com/ds-sebastian/if_curator

In some testing it worked really well getting images where I made different facial expressions, etc.