feat: add multimodal engagement estimation and adaptive tutoring

[cldanass]

Multimodal Engagement Estimation & Adaptive Tutoring

Project: Multimodal LLM-based estimation and adaptation of learner engagement
Platform: Open TutorAI CE (standalone FastAPI build)
Branch: feature/multimodal-engagement

This document explains the full feature: what it does, how it is built, how to run
it, how it was evaluated, and the results.

---

1. What this feature does

LLM tutors are "blind": they only see the text of a message. They cannot tell
whether a learner is engaged, confused, or disengaged. This feature adds an
external module that estimates the learner's engagement in real time from
three complementary modalities — text, video (webcam), and audio (voice) — fuses
them into a single engagement score, and injects a pedagogical directive into the
LLM's system prompt so the tutor adapts its behavior (simplify, check
understanding, deepen) — without any fine-tuning.

---

2. Architecture overview

A real-time pipeline in four stages, plus persistence:

Browser capture ──► Per-modality extraction ──► Fusion ──► Adaptation (LLM prompt)
 (text/webcam/mic)     (pure functions)         (weighted)    (system-prompt directive)
                            │
                            ▼
                      In-memory cache ──► SQLite (engagement.db)

Layered design (keeps computation isolated from infrastructure):

pure functions  →  cache  →  service  →  repository  →  router

Code lives in ai/engagement/:

File	Responsibility
text_core.py	Text descriptors and text score
video_core.py	Facial descriptors (MediaPipe + DeepFace) and video score
audio_core.py	Prosodic descriptors (librosa) and audio score
fusion.py	Weighted normalized fusion of present modalities
cache.py	In-memory per-user score cache (EMA smoothing, decay, TTL eviction)
service.py	Orchestration: compute → fuse → persist; engagement levels
prompt.py	Maps level to a pedagogical directive for the system prompt
repository.py / models.py / database.py	Persistence layer (SQLite)

API routes are in gateway/http/routers/engagement.py.

---

3. The three modalities

3.1 Text (text_core.py)

Seven signals extracted from each message:

Signal	Meaning
length_score	Message length, tiered (>=120 -> 0.95 … <5 -> 0.10)
lexical_diversity	Type-Token Ratio weighted by length: TTR x min(1, words/15)
question_score	1.0 ("?" + question word) / 0.7 / 0.2
message_freq_score	Based on interval since previous message
continuity_score	Regularity of the exchange rhythm
activity_score	Messages in the last 5 minutes
participation_rate	Learner messages / total session messages

Combined score (content 60% / rhythm 40%):

text = 0.25·length + 0.20·lexical_diversity + 0.15·question
       + 0.10·freq + 0.10·activity + 0.10·continuity + 0.10·participation

3.2 Video (video_core.py)

MediaPipe FaceMesh (478 landmarks) + DeepFace emotion. Six descriptors:

Descriptor	Formula	Weight
eye (EAR)	clip((EAR − 0.15)/0.20)	0.25
gaze	iris offset normalized by eye width: max(0, 1 − offset×2.2)	0.20
smile (MAR)	clip((width/height − 2.0)/8.0)	0.15
head pose	0.6·horizontal + 0.4·vertical_ok	0.15
emotion	dominant emotion weighted (happy 1.0 … angry 0.1)	0.15
attention	vertical nose–eyes offset, tiered	0.10

Penalty when gaze is strongly averted: gaze < 0.3 -> video ×= 0.6.
When no face is detected, the cached score decays instead of freezing.

3.3 Audio (audio_core.py)

librosa prosody. Four descriptors:

Descriptor	Formula	Weight
pitch	min(1, std_pitch/60)	0.35
energy (RMS)	clip(mean_RMS/0.04)	0.20
silence	1 − silence_ratio (frames < −25 dB)	0.30
speech_rate	clip(mean_ZCR×6)	0.15

Penalty when mostly silent: silence < 0.3 -> audio ×= 0.5.
Browser audio (WebM/Opus) is decoded via PyAV when soundfile cannot read it; a
local faster-whisper transcription is available as an STT fallback.

---

4. Fusion and engagement levels

Normalized weighted mean over present modalities (missing ones are dropped and
the weights renormalized — "graceful degradation"):

overall = (0.40·text + 0.30·video + 0.30·audio) / (sum of present weights)

A text-only event therefore always produces a valid score. Weights are
configurable per request.

Engagement levels (data-driven calibrated thresholds):

Level	Condition
HIGH	score >= 0.69
MEDIUM	0.53 <= score < 0.69
LOW	score < 0.53

These thresholds were calibrated from the real score distribution (tertiles)
instead of arbitrary 0.40/0.70 cut-points (see §7).

---

5. Adaptive tutoring (prompt.py)

At each completion, a short directive is injected into the LLM message (compatible
with OpenAI/Ollama and streaming), preceded by a marker telling the model not to
reveal it:

• LOW — slow down and simplify, concrete example/analogy, short steps, gentle
  re-engagement question, warm tone.
• MEDIUM — keep momentum, quick comprehension check, example if abstract.
• HIGH — deepen/accelerate, introduce a challenge or follow-up.

If no signal is available (no webcam, audio, or history), no directive is
injected and the tutor behaves normally.

---

6. API endpoints (/engagement)

Method	Route	Purpose
POST	/engagement/video	Score a webcam frame, cache it
POST	/engagement/audio	Score a voice clip, fuse, persist
POST	/engagement/chat	Score a text (or text+voice) message, fuse, persist
GET	/engagement/session/{id}/summary	Recent rows + averages
GET	/engagement/session/{id}/score	Current live score and level

Metrics are persisted in a dedicated, isolated SQLite database (engagement.db),
separate from the application DB.

---

7. Privacy & ethics

• Webcam frames and audio clips are never stored — only the last numeric score
  is cached and persisted when a message is actually sent.
• Camera and microphone are optional; any modality can be disabled.
• The design follows data minimization and assumes explicit learner consent.

---

8. Evaluation (manual_check/)

Evaluation is offline: the scripts read engagement.db + a ground-truth file
(labels.csv, learner self-rating on a 1–5 Likert scale) and produce the analysis.

PY=/home/anas/miniconda3/envs/tutorai-env/bin/python3.11

$PY manual_check/analyze.py --list-sessions                    # list sessions
$PY manual_check/analyze.py --labels manual_check/labels.csv   # ablation + separability
$PY manual_check/analyze.py --benchmark                        # real-time latency
$PY manual_check/calibrate.py --labels manual_check/labels.csv # threshold calibration

See manual_check/README.md for full usage and manual_check/RESULTATS_ANALYSE.md
for the written results.

---

9. Results summary

Protocol: 8 balanced sessions (4 engaged / 4 disengaged, 2 per scenario
S1 visual / S2 textual / S3 vocal / S4 multimodal), 41 labeled measurements.

• Separability (main result): engaged 0.696 vs disengaged 0.465 ;
  Mann-Whitney U = 3.0, p < 0.001 (statistically significant).
• Ordinal validity: Spearman rho = 0.873 (threshold-independent).
• Trimodal ordinal agreement: QWK = 0.791; errors only between adjacent
  levels.
• Threshold calibration: accuracy 0.317 -> 0.659, QWK 0.486 -> 0.791.
• Real-time latency: text ~0 ms, audio 1.5 ms, video ~185–246 ms (~4–5 fps).

Perspective: a larger, multi-user dataset will further quantify each modality's
marginal contribution and consolidate the calibration; evaluating the pedagogical
impact of the adaptation is the natural next step.

---

10. How to run the system

# Backend (conda environment)
PY=/home/anas/miniconda3/envs/tutorai-env/bin/python3.11
$PY run.py                      # starts the standalone FastAPI app

# Tests
$PY -m pytest tests/ -q                       # full suite
$PY -m pytest tests/test_engagement.py -q     # engagement module only

Note: after any change to the engagement code, restart the backend — the code
is loaded at startup, not hot-reloaded.

---

11. Task checklist (what was done)

• Text engagement module (7 signals) + calibration
• Video module (MediaPipe FaceMesh + DeepFace, 6 descriptors, gaze + decay)
• Audio module (librosa prosody, WebM/Opus decoding, STT fallback)
• Weighted normalized fusion with graceful degradation
• Adaptive directive injection into the LLM system prompt
• Dedicated SQLite persistence + isolated engagement DB
• REST API (/engagement/*) integrated into the gateway
• Front-end capture (webcam component, voice recording, stores, API client)
• Evaluation tooling (analyze, calibrate, benchmark) + labeled dataset
• Data-driven threshold calibration (0.53 / 0.69) applied across the system
• Full test suite passing
• (Perspective) Expand dataset to multi-user / graded ratings
• (Perspective) Measure pedagogical impact (baseline vs adaptive)

---

12. Additional informations :

• Engagement levels use data-driven calibrated thresholds (LOW < 0.53, HIGH >= 0.69).
• Evaluation: separability of engaged vs disengaged is statistically significant
  (Mann-Whitney p < 0.001); ordinal validity Spearman rho = 0.873; real-time latency
  validated (text/audio instantaneous, video ~4-5 fps).
• New dependencies: librosa==0.11.0, mediapipe==0.10.14, deepface (optional;
  pulls in TensorFlow — video scoring degrades gracefully when absent).

[Copilot]

Pull request overview

Adds a new multimodal “engagement” domain (text/audio/video scoring + fusion) with a dedicated SQLite DB and API endpoints, and wires the live engagement signal into chat completions via adaptive system-prompt injection. It also updates the UI to capture webcam frames and attach voice + webcam context to engagement events, plus adds evaluation tooling and tests.

Changes:

• Introduce ai/engagement/ (scoring cores, fusion, cache, persistence, prompt injection) backed by an isolated engagement SQLite DB.
• Add /api/v1/engagement/* routes, initialize engagement DB on app startup, and inject engagement directives into provider proxy chat bodies when enabled.
• Update UI to capture webcam engagement, send engagement events for text/voice, and add offline evaluation scripts + new tests.

Reviewed changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
ui/vite.config.ts	Makes dev proxy target configurable via VITE_BACKEND_URL.
ui/src/lib/stores/index.ts	Adds a store for the live webcam engagement score.
ui/src/lib/components/chat/WebcamCapture.svelte	New webcam capture + periodic scoring component publishing a live score.
ui/src/lib/components/chat/MessageInput/VoiceRecording.svelte	Changes voice confirm flow to include base64 audio payload for engagement.
ui/src/lib/components/chat/MessageInput.svelte	Records engagement on text/voice send; adds engagement camera toggle and headless capture.
ui/src/lib/apis/engagement/index.ts	New frontend API client for engagement endpoints.
tests/test_media.py	Adds test coverage for local Whisper STT fallback.
tests/test_engagement.py	Adds fusion/service/router/prompt-injection tests for engagement.
tests/conftest.py	Adds isolated in-memory engagement DB wiring for tests.
run.py	Adds a minimal app entrypoint for local dev (uvicorn run:app).
requirements.txt	Adds engagement-related dependencies (e.g., librosa, mediapipe).
manual_check/sortie_benchmark.md	Adds benchmark output artifact for engagement latency.
manual_check/RESULTATS_ANALYSE.md	Adds written analysis/results summary for engagement evaluation.
manual_check/README.md	Documents manual evaluation tooling and dataset format.
manual_check/labels.csv	Adds sample ground-truth labels file used by evaluation scripts.
manual_check/grid_search.py	Adds fusion-weight grid search tooling.
manual_check/calibrate.py	Adds threshold calibration tooling for engagement levels.
manual_check/analyze.py	Adds core evaluation script (ablation, correlations, separability, benchmark).
manual_check/ab_analyze.py	Adds A/B analysis tooling (adaptive vs control).
learning/sessions/service.py	Notes engagement is recorded client-side (no backend hook here).
gateway/http/routers/providers.py	Injects engagement directive into proxied OpenAI/Ollama chat requests (guarded by setting).
gateway/http/routers/engagement.py	Adds engagement API router (video/audio/chat + session score/summary).
gateway/http/routers/audio.py	Adds local faster-whisper STT fallback when no external STT URL is configured.
gateway/http/app.py	Initializes engagement DB on startup and registers engagement router.
config/settings.py	Adds ENGAGEMENT_ADAPTIVE_PROMPT setting to gate adaptive prompt injection.
ai/media/whisper_stt.py	Implements local STT via faster-whisper with lazy model loading.
ai/engagement/video_core.py	Implements video engagement scoring (MediaPipe FaceMesh + optional DeepFace).
ai/engagement/text_core.py	Implements text engagement metrics/scoring with session-scoped cache.
ai/engagement/service.py	Orchestrates scoring, fusion, caching, and persistence.
ai/engagement/repository.py	Adds repository methods for recent rows and averages.
ai/engagement/prompt.py	Builds/injects engagement directive into system prompt based on live cached signals.
ai/engagement/models.py	Adds ORM model for engagement metrics table (dedicated DB).
ai/engagement/fusion.py	Implements normalized weighted fusion over present modalities.
ai/engagement/database.py	Adds dedicated DB engine/session + init logic for engagement DB.
ai/engagement/cache.py	Adds runtime caches (EMA smoothing, TTL eviction, freshness window).
ai/engagement/audio_core.py	Implements audio engagement scoring with soundfile + PyAV fallback decode.
ai/engagement/init.py	Exposes engagement service/repository as a domain package.

[cldanass]

• Engagement DB migration safety (ai/engagement/database.py): the incompatible-schema path no longer drops the engagement_metrics table automatically. It now raises and requires an explicit ENGAGEMENT_DB_ALLOW_DROP=true opt-in, preventing silent data loss on startup.

• Data minimization (ai/engagement/service.py): raw message text is no longer persisted. Only engagement scores and word counts are stored, which aligns the implementation with the stated privacy design.

• Voice auto-send (ui/src/lib/components/chat/MessageInput.svelte): transcription now auto-sends only when the user has enabled speechAutoSend, instead of always sending.

• Webcam capture guard (ui/src/lib/components/chat/WebcamCapture.svelte): added an in-flight guard so a slow capture skips the next tick rather than stacking concurrent frame requests.

The corresponding review threads have been resolved.