This project allows you to play web-based games (specifically configured for "Steve's World") using hand gestures captured by your webcam. It combines a webview for the game and a background process for gesture recognition.
- Gesture Control: Control the game character using hand movements.
- Integrated Browser: Launches the game directly in a dedicated window.
- Real-time Tracking: Uses MediaPipe and OpenCV for fast and accurate hand tracking.
You can play various side-scrolling or platformer games. Here are some recommended ones:
To switch games, edit the url parameter in app.py.
The following hand gestures are mapped to game actions:
| Gesture | Action |
|---|---|
| Pinky Finger Up | Move Right |
| Index Finger Up | Move Left |
| All Fingers Up | Jump |
| Pinky + Ring Fingers Up | Jump + Move Right |
| Index + Middle Fingers Up | Jump + Move Left |
- Python 3.x
- A working webcam
-
Clone the repository or download the source code.
-
Install the required Python packages:
pip install opencv-python mediapipe pyautogui pywebview
-
Play Steve's World Run
app.pyto play "Steve's World" as it is defined in the project. This launches the game in a dedicated window with controls ready to go.python app.py
-
Universal Game Control Run
gesture_control.pyto use the gesture controls for any other game (like Mario) running in your browser. This starts the camera tracking so you can control any game that uses the mapped keys.python gesture_control.py
app.py: The main entry point. It launches apywebviewwindow to display the game and starts the gesture control script as a subprocess. It handles the lifecycle of the application, ensuring the gesture controller stops when the window is closed.gesture_control.py: Captures video frames, processes them with MediaPipe to detect hand landmarks, and simulates keyboard presses using PyAutoGUI based on the detected gestures.