Switch to @vladmandic/human

[ghisch]

This PR updates the face detection classifier from @vladmandic/face-api to @vladmandic/human.
Fix #1171

Key Changes

1. Improved Performance

   • @vladmandic/human offers significantly better performance compared to face-api, with faster processing and more reliable face detection.
   • It detects more faces per image
   • Unexpectedly identifies pet faces (tested with my dog 🐶).

2. Configuration Adjustments

   • Updated maxDetected to 100 (default is 1) to support multiple faces in a single image.
   • Face vector: previously called descriptor (128 dimensions in face-api), now called embedding (1024 dimensions in human).

3. Face Clustering

   • Face clustering does not work properly right now. It's limited to handling duplicate images. I don't know why. I feel it's linked to the dimensions size difference.
   • Clustering parameters (MIN_CLUSTER_SEPARATION, MAX_CLUSTER_EDGE_LENGTH) might need optimization for the new 1024-dimensional embeddings. Suggestions and assistance on tuning these are welcome! 🙏

4. Compatibility Note

   • Breaking Change: Existing detections made with face-api are incompatible with human embeddings. This requires a reset of existing data to align with the new system.

Testing

• Tested on my LSIO Nextcloud instance in PURE_JS (WASM) mode only.
• Needs testing in GPU, CPU, and Node.js environments.

Request for Feedback

• I'm not a developer and relied on ChatGPT for much of the implementation. Feedback and suggestions for improvement are highly appreciated.

Recap (Help Needed 🙏)

• Optimize face clustering to accommodate the higher-dimensional embeddings.
• Verify functionality across GPU, CPU, and Node.js environments.

Let me know if there's anything I can improve or clarify further!

[Felitendo]

Thank you sooo much for this PR, I think this is strongly needed. The face-api Repo hasn't gotten a meaningful commit since January and is also Archived now. Hopefully @marcelklehr can look at this PR when he has time :)

[marcelklehr]

Hello @ghisch
Thank you for taking the time to compose this PR. Indeed using a new embedding model needs careful calibration of the hyperparameters and extensive testing. I can currently not provide either in my capacity as Nextcloud GmbH engineer, sadly. I can try to look into this in my free time, but can't be sure when I'll get to this.

[ghisch]

Hello @marcelklehr 👋 I totally understand no problem.

Also, FYI, human lib repo also provide some kind of clustering in JS, I feel like inspiration can be taken from here.

• https://github.com/vladmandic/human/tree/main/demo/facematch
• https://github.com/vladmandic/human/wiki/Embedding#face-recognition

[github-actions]

Hello there,
Thank you so much for taking the time and effort to create a pull request to our Nextcloud project.

We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process.

Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6

Thank you for contributing to Nextcloud and we hope to hear from you soon!

(If you believe you should not receive this message, you can add yourself to the blocklist.)

[QuentinLemCode]

Hello @marcelklehr , had you had the time to take a look on this PR ?
Face recognition needs an upgrade ;)

I can help to make the CI pass

[marcelklehr]

@QuentinLemCode My earlier comment still applies, sadly.

[marcelklehr]

@QuentinLemCode My earlier comment still applies, sadly. This needs testing in normal CPU mode and with GPU, Face clustering needs to be fixed. If you can help, by all means :)

[QuentinLemCode]

@QuentinLemCode My earlier comment still applies, sadly. This needs testing in normal CPU mode and with GPU, Face clustering needs to be fixed. If you can help, by all means :)

@marcelklehr Sure! Do you have any documentation or information for launching the local dev env and performing tests?

[marcelklehr]

Here are our tutorials for developers: https://nextcloud.com/developer/

[QuentinLemCode]

@marcelklehr In the CI, a job has failed because an archive of photos isn't available anymore on your site

Could you rehost it or send it here please ? I suggest to include it in a test assets folder

[marcelklehr]

Could you rehost it or send it here please ? I suggest to include it in a test assets folder

I've rehosted it, thanks for the heads up!

[MB-Finski]

Oh, this is really interesting! I can help with the hyper-parameter optimization since I'm somewhat familiar with the clustering algo. The face-api embedding model is quite old so I'm really interested in seeing how much this could improve the clustering results. However, more dimensions alone isn't necessarily better for clustering due to what is called "the curse of dimensionality". As the dimensionality of the vectors increases the distances between any two random vectors become more narrowly distributed making separating noise from a valid signal more difficult.

Be that as it may: Any trace as to why the clustering fails? I take it there's no error in the logs? @ghisch: By 'limited to duplicate images', do you mean that only faces in duplicate images get clustered together?

The clustering algorithm itself doesn't care about the dimensions as long as a distance calculation between two embeddings can be done so there should be no reason it wouldn't work for 1024 dimensions per se (although I expect it to be exactly 8 times slower than for 128 dimensions).

[ghisch]

Any trace as to why the clustering fails? I take it there's no error in the logs?

As I said I'm not a dev and mainly relied on AI to propose this PR. But from what I understood, it was creating one cluster per face detected. No error (but also I was not running a dev instance, I was using my own production instance).

By 'limited to duplicate images', do you mean that only faces in duplicate images get clustered together?

When the same image file was duplicated (and/or slightly edited like cropped), all faces were correctly clustered.

[marcelklehr]

Thank you for your work on this @ghisch 💙 We appreciate your effort and the work you've put into this PR.
I'd like to invite you to our Developer Community Chat. Hit me up if you would like a guest account for that instance!

I wanted to inform you that we are planning to port this application to Python. Due to this significant change, the modifications proposed in your pull request will no longer be relevant, and we have decided not to pursue this PR further.

We hope you understand our decision, and we encourage you to contribute to the project once we start working on the Python version.

Thank you for your understanding and continued support.

[ghisch]

Ok thanks for the heads-up @marcelklehr . May I ask:

1. What's the expected timeline if any ?
2. Is the technology already chosen ?
3. Will you only port face detection to python ? Or will you also port face clustering ?

Thanks 🙏

[marcelklehr]

Thanks for understanding. The timeline is some time this year. The full stack is not chosen yet, we're likely going to make use of AppAPI to build this as an ExApp, likely in python, because it lends itself to ml tasks. We're not yet sure if we'll port the clustering as well, yet.

[ghisch]

We're not yet sure if we'll port the clustering as well, yet.

Strongly suggest to port both face detection and clustering. Most face detection technology actually provides some kind of clustering capability. Plus having both features on the same stack makes it easy to maintain, upgrade and understand.

Thanks for the heads-up, can't wait to see some needed improvements on recognize :)