Suggestion: Include sample for Facemark?

[alastaira]

It would be nice to include a sample of the Facemark face landmark detector, as there seems to be no examples to be found on the web anywhere! Trying to translate the example from the OpenCV source into OpenCvSharp seems to lead to a crash when calling fit(), so any expertise you can offer in terms of how this should be done would be much appreciated!

Here's my current (broken) code:

'''
// Load the source image and convert to greyscale
Mat input = new Mat(Path.Combine(Application.streamingAssetsPath, "aa.png"), ImreadModes.Color);
Mat grey = new Mat();
Cv2.CvtColor(input, grey, ColorConversionCodes.BGR2GRAY);
// This works, so the greyscale image is being created ok
// Cv2.ImShow("Greyscale", grey);

// Find faces in the image
var classifier = new CascadeClassifier(Path.Combine(Application.streamingAssetsPath, "haarcascade_frontalface_alt.xml"));
OpenCvSharp.Rect[] facesRects = classifier.DetectMultiScale(grey);

// This works, so faces are being found by the HAAR detector
// Debug.Log(string.Format("{0} faces detected:", facesRects.Length));
// for (int i = 0; i < facesRects.Length; i++) {
//`Debug.Log(facesRects[i].ToString());
//}

// Create a new facemark instance and load the model
OpenCvSharp.Face.Facemark facemark = OpenCvSharp.Face.FacemarkLBF.Create();
facemark.LoadModel(Path.Combine(Application.streamingAssetsPath, "lbfmodel.yaml"));

// Create inputs for the fit function
InputArray facesArray = InputArray.Create(facesRects);
Mat landmarks = new Mat();

// Following line causes a crash :(
facemark.Fit(grey, facesArray, landmarks);

[0x0aNL]

I've managed to get facemark.Fit to work using 4.1.0, but with some editing involved. I'm quite sure it's not the right way to do it, but at least it works.

1. The opencv_contrib module for FacemarkLBF got an update (opencv/opencv_contrib@4593e63#diff-cdc108db65c8decfc40762a6462dd8f7) that seems to make it more careful about memory allocation (as far as I understand). I rebuilt OpenCV 4.1.0 with opencv_contrib at that commit.
2. I changed OpenCVSharp to expect Point2f[][] instead of InputOutputArray for facemark.Fit:

• ./OpenCvSharpExtern/face_Facemark.h: std::vector< std::vector<cv::Point2f> > *landmarks instead of cv::_OutputArray *landmarks). Recompile OpenCVSharp after this change

• OpenCvSharp/Modules/face/Facemark/Facemark.cs:

   `public virtual bool Fit(
        InputArray image,
        InputArray faces,
        out Point2f[][] landmarks)
   {
        ThrowIfDisposed();
        if (image == null)
            throw new ArgumentNullException(nameof(image));
        if (faces == null)
            throw new ArgumentNullException(nameof(faces));
        image.ThrowIfDisposed();
        faces.ThrowIfDisposed();`
  
        `int ret;
        using(var landmarx = new VectorOfVectorPoint2f()) { 
            ret = NativeMethods.face_Facemark_fit(ptr, image.CvPtr, faces.CvPtr, landmarx.CvPtr);
            landmarks = landmarx.ToArray();
        }
  
        GC.KeepAlive(this);
        GC.KeepAlive(image);
  
        return ret != 0;
    }`
  

Now, the function can be called with:

            OpenCvSharp.Rect[] faces = classifier.DetectMultiScale(grey);
            InputArray facesArray = InputArray.Create(facesRects);
            Point2f[][] landmarks;
            facemark.Fit(grey, facesArray, out landmarks);