Mediapipe Based Gesture Recognition

[kerem0comert]

Hello,

With the hope that this is the appropriate place to ask such a question, I would like your opinions about choosing the ideal pyG based model for my purposes and ideas in general that would lead me to the right direction.

I would like to perform Gesture Recognition, based on Mediapipe Hands.

To give you an idea, some of my inputs are as follows and I need to classify each image to a Gesture class (Like Fist, Open Hand, Pinky etc.):

One property of my graphs is that all graphs are guaranteed to have exactly 21 nodes (which correspond to hand landmarks) and 21 edges (which corresponds to joints) – assuming that none of the landmarks are obscured. The ordering of the nodes is also not trivial, each node corresponds to a specific landmark and the edges always connect to the same nodes—thus the adjacency matrices for each image is the same.
However, the geometry of the overall graph is crucial in the detection of the gesture.

The classical methods work quite well so far, given 30-40 images per each class as training data. But since I want to perform one-shot or multi-shot learning, this lead me to believe that GNNs would be more fit to the job.

Looking forward to suggestions, and I can elaborate further if necessary.

[rusty1s]

I always suggest SplineConv for these kind of problems. It can handle these spatial graphs quite well. See examples/faust.py for a basic implementation.