Gesture Recognition Actions

Real-time hand gesture recognition and automation system using OpenCV, Python, and machine learning (CNN & SVM).

Features

Modern Architecture

• Clean, modular code structure
• Proper error handling and logging
• Configuration management
• Type hints for better IDE support

Multiple Recognition Modes

• SVM Mode: Real-time gesture detection for action automation
• Segmentation Mode: Hand tracking and finger counting

Machine Learning

• CNN (Convolutional Neural Network) for gesture classification
• SVM (Support Vector Machine) for gesture detection
• Easy model training and evaluation

Performance

• Frame downsampling for faster processing
• FPS counter and monitoring
• Multi-threaded camera reading
• Optimized detection pipeline

Automation

• Automatic keyboard actions
• Mouse control integration
• Customizable gesture-to-action mapping
• Action cooldown to prevent spam

Project Structure

gesture-recognition-actions/
├── config.py                  # Centralized configuration
├── utils.py                   # Helper utilities and classes
├── train_cnn.py              # CNN model training
├── gesture_detector.py        # SVM-based gesture detection
├── hand_segmentation.py      # Hand segmentation utilities
├── app.py                    # Main application
├── preprocessing/            # Legacy preprocessing scripts
├── models/                   # Trained models
├── datasets/                 # Training datasets
├── requirements.txt          # Python dependencies
└── README.md                # This file

Installation

Prerequisites

• Python 3.8 or higher
• Webcam connected to your computer
• macOS, Linux, or Windows

Setup

1. Clone the repository

cd /path/to/gesture-recognition-actions

2. Create virtual environment (recommended)

python3 -m venv venv
source venv/bin/activate  # macOS/Linux
# or
venv\Scripts\activate  # Windows

3. Install dependencies

pip install -r requirements.txt

Configuration

Edit config.py to customize:

Camera Settings

CAMERA_ID = 0                    # Camera device ID (0=default, 1=external)
CAMERA_WIDTH = 1280
CAMERA_HEIGHT = 720
SCALE_FACTOR = 4                 # For downsampling frames

Gesture Actions

Configure gesture-to-action mapping:

GESTURE_ACTIONS = {
    "Pause": ("press", "space"),
    "Scrolling Up": ("scroll", -7),
    "Scrolling Tabs": ("hotkey", ("ctrl", "pgup")),
    "Change Program": ("hotkey", ("alt", "tab")),
}

Detection Thresholds

SVM_CONFIDENCE_THRESHOLD = 0.90      # 90% minimum confidence
COOLDOWN_MS = 500                     # Milliseconds between actions

Usage

1. Training CNN Model

To train a new CNN gesture model:

python train_cnn.py

Requirements:

• Dataset organized in datasets/Below_CAM/ with subdirectories per gesture
• Directory structure: datasets/Below_CAM/A/, datasets/Below_CAM/C/, etc.

Configuration:

CNN_EPOCHS = 20
CNN_BATCH_SIZE = 32
CNN_IMAGE_SIZE = (300, 300)

2. Real-time Gesture Recognition (SVM Mode)

Detect hand gestures and perform actions:

python app.py --mode svm --camera 0

Features:

• Real-time gesture detection
• Automatic action execution
• FPS monitoring
• Confidence score display

Controls:

• Press 'q' to quit the application

3. Hand Segmentation Mode

Track hand and count fingers:

python app.py --mode segment --camera 0

Features:

• Automatic background calibration (30 frames)
• Finger counting
• Hand contour visualization
• Hand size and center tracking

API Reference

GestureRecognitionApp

Main application class:

from app import GestureRecognitionApp

# Create app instance
app = GestureRecognitionApp(use_svm=True)

# Setup and run
app.setup()
app.run(mode="svm")
app.cleanup()

# Or use context manager
with GestureRecognitionApp() as app:
    app.run(mode="svm")

SVMGestureDetector

Gesture detection using SVM models:

from gesture_detector import SVMGestureDetector

detector = SVMGestureDetector()

# Detect gestures in frame
detections = detector.detect_gestures(frame, scale_factor=4)

# Draw detection boxes
frame = detector.draw_detections(frame, detections)

# Perform actions
for detection_dict, gesture_idx, confidence in detections:
    gesture_name = detector.gesture_names[gesture_idx]
    detector.perform_action(gesture_name)

HandSegmenter

Hand segmentation and finger counting:

from hand_segmentation import HandSegmenter, ROI, process_frame_for_segmentation

segmenter = HandSegmenter()
roi = ROI()

# Process frame
roi_frame = process_frame_for_segmentation(frame, roi)

# Update background (calibration phase)
segmenter.update_background(roi_frame)

# Segment hand
result = segmenter.segment_hand(roi_frame)
if result:
    thresholded, hand_contour = result
    
    # Count fingers
    finger_count = segmenter.count_fingers(thresholded, hand_contour)

CameraManager

Manage webcam operations:

from utils import CameraManager

# Manual usage
camera = CameraManager(camera_id=0, width=1280, height=720)
camera.open()

success, frame = camera.read_frame()
camera.release()

# Context manager (recommended)
with CameraManager(camera_id=0) as camera:
    success, frame = camera.read_frame()

FpsCounter

Monitor frames per second:

from utils import FpsCounter

fps_counter = FpsCounter(update_interval=30)

while True:
    fps = fps_counter.update()
    print(f"Current FPS: {fps:.2f}")

Troubleshooting

Camera Not Found

# Check available cameras
import cv2
for i in range(5):
    cap = cv2.VideoCapture(i)
    if cap.isOpened():
        print(f"Camera {i} is available")
    cap.release()

# Use correct camera ID in config
config.CAMERA_ID = 1  # or your camera ID

Model Not Loading

ERROR: File not found: models/Pause_detector.svm

• Ensure SVM models are in the models/ directory
• Download pre-trained models or train your own

Poor Detection Performance

1. Improve lighting - Ensure adequate, consistent lighting
2. Train custom models - Use your own gesture data
3. Adjust threshold - Lower SVM_CONFIDENCE_THRESHOLD for sensitivity
4. Scale factor - Use SCALE_FACTOR = 2 for faster but less accurate detection

Actions Not Executing

• Check config.GESTURE_ACTIONS mapping
• Verify gesture confidence is above SVM_CONFIDENCE_THRESHOLD
• Check action cooldown: COOLDOWN_MS
• Ensure PyAutoGUI has permission (macOS may require accessibility permissions)

Performance Optimization

For Real-time Performance

# In config.py
SCALE_FACTOR = 4                           # More downsampling = faster
SVM_UPSAMPLE_NUM_TIMES = 0                 # Less upsampling = faster
DISPLAY_FPS = True                         # Monitor FPS

For Higher Accuracy

SCALE_FACTOR = 2                           # Less downsampling = more accurate
SVM_CONFIDENCE_THRESHOLD = 0.95            # Higher threshold

Machine Learning Models

CNN Training

• Architecture: 3 Convolutional layers + 2 Dense layers
• Input: 300x300 grayscale images
• Output: 17 gesture classes
• Loss: Sparse categorical crossentropy
• Optimizer: Adam

SVM Models

• Feature: HOG (Histogram of Oriented Gradients)
• Type: dlib FHOG object detector
• Training: One-vs-All classification

Dependencies

• opencv-python - Computer vision
• tensorflow - Deep learning framework
• scikit-learn - Machine learning utilities
• dlib - Object detection (SVM)
• numpy - Numerical computing
• pyautogui - GUI automation
• imutils - Image utilities
• matplotlib - Visualization

Development

Code Style

• PEP 8 compliant
• Type hints throughout
• Comprehensive docstrings
• Proper logging

Adding New Gestures

1. Collect training data

   • Create subdirectory in datasets/Below_CAM/NEW_GESTURE/
   • Add gesture images

2. Update config

   CNN_GESTURE_NAMES = ["A", "C", ..., "NEW_GESTURE"]

3. Train model

   python train_cnn.py

4. Update gesture actions (optional)

   GESTURE_ACTIONS["NEW_GESTURE"] = ("hotkey", ("alt", "n"))

Logging

Application logs are written to gesture_recognition.log:

2024-01-15 10:30:45,123 - app - INFO - Application setup completed
2024-01-15 10:30:46,456 - gesture_detector - INFO - Loaded model: pause from models/Pause_detector.svm

Configure logging in config.py:

LOG_LEVEL = logging.INFO
LOG_FORMAT = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
LOG_FILE = "gesture_recognition.log"

Benchmarks

On MacBook Pro (M1):

• SVM Detection: ~45-60 FPS (with 4x downsampling)
• Finger Counting: ~30-40 FPS
• CNN Prediction: ~100 FPS (GPU-accelerated with TensorFlow)

License

MIT License - see LICENSE file for details

Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

Author

Lotfi Habbiche - Software Engineer

• Email: habbichelotfi@gmail.com
• GitHub: habbichelotfi

Support

For issues and questions:

1. Check the Troubleshooting section
2. Review the API Reference
3. Check logs in gesture_recognition.log
4. Open an issue on GitHub

Acknowledgments

• OpenCV for computer vision
• TensorFlow/Keras for deep learning
• dlib for object detection
• scikit-learn for ML utilities