Machine learning - how it works, Qs

[weitheng]

I am very impressed by Immich’s facial recognition capabilities. Can someone please enlighten me how it works - what ML model is being used, is it possible to use a different model or export the model etc. I have read the documentation about the architecture, but they are very much on the surface.

[mertalev]

The facial recognition models we use are from InsightFace. The process looks something like this:

1. Generate preview image
2. Send preview image to machine learning service
3. Machine learning service preprocesses image (resize, crop, normalize)
4. Detection model detects bounding boxes of each face and their landmarks (eyes, nose, mouth)
5. Filter out faces below the detection threshold
6. For each detected face, use their landmarks to align the faces so they're not at an angle
7. For each detected face, use recognition model to generate embedding from aligned face image
8. Respond with bounding boxes and embeddings of each detected face
9. Add each face to the database
10. Queue each face for clustering

• Clustering only starts when there are no more active thumbnail or face detection jobs

11. Cluster all faces in the queue with a modified version of the DBSCAN algorithm
    a. For each face, fetch the most similar faces below the distance threshold
    b. If there are not as many faces as the min recognized faces setting, move this face to the end of the queue and move on
    c. If there are, then check if any of these faces have associated people
    d. If they do, assign the person of the most similar face to this face
    e. If they don’t, create a new person, assign this face to that person and create a person thumbnail based on this face
    f. At the end, the deferred faces without enough matches are retried.
    g. If there are any assigned people among the faces they found, they’re assigned the person of the most similar face.
    h. If not, they’re treated as outliers and cannot create new people.

[weitheng]

Thank you so much for the detailed response! Looking at this, the default model we are using from InsightFace is buffalo_l?

Is there a way for us to export the trained model?

[libkush]

I think a pre-trained model from InsightFace is being used. You can take a look here to see all pre-trained models.