A MediaPipe project that allows anyone to write a sentence using gestual language from the LGP (Linguagem Gestual Portuguesa) easily. Custom gestures can be made without knowledge of MediaPipe or Python, just by using a simple UI.
Run via local Python:
cd src
pip install -r requirements.txt
python3 main.pyRun via docker:
docker compose buildOpen the app.
python3 main.pyHow to create a new gesture:
Inside the UI, press the "C" button to create a new gesture using our Gesture Creator UI.If you want to create a new gesture using code, head to src/gestures/, create a new gesture inside letters, numbers, saudations or verbs and add it to the dictionary on lgp_recognition.py.
Run via docker:
docker compose buildThis project was done in a university course, so it is not production ready. It is a proof of concept that MediaPipe can be used to create a gestual language detection and sentence building app. If you wish to research this matter, feel free to contact us to collaborate or ask any questions.
For future development, the group envisions using embeddings to enhance gesture recognition. Rather than relying on manually defined distances between finger landmarks, this approach would represent each gesture as a single vector automatically learned by a model. To achieve this, 21 hand landmarks (each with x, y, z coordinates) would be extracted per frame, resulting in a 63-dimensional input vector. This data would be normalised (e.g. centred around the wrist and scaled), then passed into a model that transforms it into a compact embedding representing the overall gesture. This method has the potential to make the system more scalable, adaptable, and robust to variations across different users and conditions.
- João Ferreira @Xurape
- Hugo Guimarães @Guimaraes04