Skip to content

README.md

Latest commit

 

History

History
154 lines (105 loc) · 5.41 KB

README.md

File metadata and controls

154 lines (105 loc) · 5.41 KB

Fitsense Card

A mobile app that uses motion sensor data and a hybrid CNN-LSTM model to automatically recognize physical exercises in real time, with a human-in-the-loop feedback system that continuously improves predictions.

📹 Watch the Demo on YouTube

Table of Contents

Overview

FitSense is a React Native mobile application that automatically detects and classifies physical exercises by analyzing accelerometer and gyroscope data from the device's motion sensors. Users simply start a session, work out, and the app predicts what exercise they are performing, no manual logging required.

The core ML model is a hybrid 1D CNN-LSTM architecture trained on the mHealth dataset, capable of recognizing 12+ exercise types. Inference runs on a cloud-deployed API, delivering predictions within a few seconds of sensor data collection.

What makes FitSense distinctive is its human feedback loop: when the model makes a wrong prediction, users can correct it. That feedback is captured and used to improve the model over time, which turns every user interaction into a training signal.

Key Features

  • Automatic Exercise Recognition — classifies 12+ exercise types from raw sensor data
  • Real-Time Predictions — live inference powered by a cloud-deployed ML API
  • Human Feedback Loop — users correct wrong predictions, feeding a continuous improvement cycle for the model
  • Session History & Stats — full workout logs, exercise history, and personal statistics
  • Badge System — gamified achievements to keep users engaged
  • Firebase Authentication — secure user management

Architecture

The system is composed of a React Native frontend and a Firebase backend handling authentication, model inference, and data storage.

Architecture Diagram

Frontend — React Native (Expo)

  • Collects motion sensor data (accelerometer + gyroscope) during a session
  • Sends sensor payloads to the backend API for inference
  • Displays predicted exercise to the user in real time
  • Captures user feedback when a prediction is incorrect

Backend — Firebase

Service Role
Firebase Authentication User credentials and auth tokens
Cloud Run (Deployed API) Loads ML model, runs inference, returns predictions
Cloud Storage Stores the trained PyTorch model artifact
Firestore Stores predictions, session logs, user stats, badges, and feedback

The Model

The exercise classifier is a hybrid 1D CNN-LSTM model built with PyTorch:

  • CNN layers extract local temporal patterns from the raw sensor signal windows
  • LSTM layers capture longer-range sequential dependencies across the time series
  • Trained on the mHealth dataset — a public benchmark for activity recognition from body-worn sensors
  • 12+ exercise classes recognized, including activities like walking, running, cycling, and strength exercises

The model is exported and served via a Firebase Cloud Run API, decoupling inference from the mobile client and keeping the app lightweight.

Human Feedback Loop

One of the core design decisions in FitSense is treating user corrections as first-class data.

When the model predicts the wrong exercise, the user can correct it directly in the app. This correction is:

  1. Stored in Firestore alongside the original sensor data
  2. Used as labeled training data to retrain and improve the model in future iterations

This creates a continuous improvement cycle, the more the app is used, the more accurate it becomes. It also means the model can adapt to individual users' sensor patterns over time, addressing one of the key challenges in activity recognition: variability across devices and users.

Tech Stack

Layer Technology
Mobile Frontend React Native (Expo)
Backend & Auth Firebase (Firestore, Auth, Cloud Storage)
ML Inference API Firebase Cloud Run
Model Training Python, PyTorch
Dataset mHealth Dataset

Getting Started

Prerequisites

  • Node.js and npm
  • Expo CLI: npm install -g expo-cli
  • A Firebase project with Authentication, Firestore, and Storage enabled

Setup

  1. Clone the repository

    git clone https://github.com/JoaoCoelho2003/SA-FitSense.git
cd SA-FitSense

  2. Install frontend dependencies

    cd frontend
npm install

  3. Configure Firebase

    Create a firebaseConfig.js file inside frontend/ with your Firebase project credentials:

    export const firebaseConfig = {
  apiKey: "YOUR_API_KEY",
  authDomain: "YOUR_AUTH_DOMAIN",
  projectId: "YOUR_PROJECT_ID",
  storageBucket: "YOUR_STORAGE_BUCKET",
  messagingSenderId: "YOUR_MESSAGING_SENDER_ID",
  appId: "YOUR_APP_ID"
};

    Download your google-services.json from the Firebase Console (Project Settings → Your Apps) and place it in the frontend/ folder. This file is gitignored and not included in the repo.

  4. Start the app

    expo start

Running the API locally

cd fitsense-api
pip install -r requirements.txt
python app.py