This document provides a comprehensive overview of the Cue project for AI assistants like Claude to understand the codebase, architecture, and implementation details.
- Project Overview
- Architecture
- Directory Structure
- Technology Stack
- Core Components
- API Endpoints
- Data Flow
- Key Features
- Design Principles
- Setup & Configuration
- Code Patterns
Cue (formerly REMIND-AR) is a dementia care assistant that provides real-time face recognition and memory support for dementia patients. The system has two distinct modes:
- Caregiver Mode: Administrative interface for enrolling people and managing profiles
- Patient Mode: Real-time recognition interface that displays contextual information and records memories
Dementia patients struggle to recognize loved ones. Cue provides:
- Visual Cues: Immediate display of name, relationship, and context when someone is recognized
- Audio Reassurance: Gentle voice cue providing comfort ("This is Rahul, your grandson...") in 5 languages
- Memory Support: Passive recording and summarization of conversations for later context
- Multi-Language: Hindi, Tamil, Bengali, Telugu support via Sarvam AI
┌─────────────────────────────────────────────────────────────┐
│ FRONTEND (React) │
│ ┌──────────────────┐ ┌─────────────────────────┐ │
│ │ Caregiver Mode │ │ Patient Mode │ │
│ │ (Setup/Admin) │ │ (Real-time Recognition) │ │
│ └──────────────────┘ └─────────────────────────┘ │
└────────────────────┬────────────────────┬──────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────┐
│ BACKEND (FastAPI) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Face Rec │ │ LLM │ │ STT │ │ TTS │ │
│ │InsightFace│ │ (Groq) │ │ (Groq) │ │ElevenLabs│ │
│ └──────────┘ └──────────┘ └──────────┘ │ + Sarvam │ │
│ └──────────┘ │
└────────┬───────────────┬──────────────────────────────────┘
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Qdrant │ │ Neo4j │
│ (Vector DB) │ │ (Graph DB) │
│ - Faces │ │ - Profiles │
│ - Memories │ │ - Memories │
└──────────────┘ └──────────────┘
-
Dual Database Strategy:
- Qdrant: Fast vector similarity search for face matching and semantic memory search
- Neo4j: Graph database for relationships, metadata, and structured data
-
Hybrid Processing:
- Local: InsightFace with ONNX (privacy, 5-10x faster than FaceNet)
- Cloud: Groq LLM, Groq Whisper, ElevenLabs (quality, scalability)
-
Real-time Loop: 300ms face recognition polling with multi-frame validation
hackathon/
├── frontend/ # React frontend
│ ├── src/
│ │ ├── pages/
│ │ │ ├── CaregiverMode.jsx # Admin panel
│ │ │ └── PatientMode.jsx # Recognition interface
│ │ ├── components/
│ │ │ ├── HUD.jsx # Heads-up display overlay
│ │ │ └── RecordButton.jsx # Audio recording controls
│ │ ├── hooks/
│ │ │ ├── useFaceTracking.js # Face recognition loop
│ │ │ └── useAudioRecorder.js # Memory recording
│ │ └── services/
│ │ └── api.js # Backend API calls
│ └── vite.config.js
│
├── backend/ # FastAPI backend
│ ├── app/
│ │ ├── main.py # FastAPI app entry
│ │ ├── config.py # Environment config
│ │ ├── models/
│ │ │ └── schemas.py # Pydantic models
│ │ ├── routers/
│ │ │ ├── recognize.py # Face recognition endpoint
│ │ │ ├── hud.py # HUD context endpoint (multi-lang)
│ │ │ ├── whisper.py # Audio cue endpoint (multi-lang)
│ │ │ ├── memory.py # Memory save endpoint
│ │ │ └── caregiver.py # CRUD for people
│ │ ├── services/
│ │ │ ├── face_recognition.py # InsightFace wrapper
│ │ │ ├── vector_db.py # Qdrant client
│ │ │ ├── graph_db.py # Neo4j client
│ │ │ ├── llm.py # Groq LLM service
│ │ │ ├── sarvam.py # Sarvam Translate + TTS
│ │ │ ├── whisper.py # Groq STT service
│ │ │ ├── elevenlabs.py # ElevenLabs TTS (English)
│ │ │ └── routine.py # Routine extraction
│ │ ├── workers/
│ │ │ └── routine_worker.py # Background routine extraction
│ │ ├── data/
│ │ │ ├── relations.py # Multi-lang relation dictionary
│ │ │ └── whisper_templates.py # Audio templates per language
│ │ └── utils/
│ │ └── image.py # Image processing
│ └── requirements.txt
│
└── face_images/ # Stored face thumbnails
- React 18 (Vite) - UI framework
- MediaPipe - Real-time face detection in browser
- Web Audio API - Microphone recording
- FastAPI - Async Python web framework
- InsightFace (buffalo_s model) - Face embedding extraction (512-dim, ONNX-accelerated)
- Groq API:
llama-3.3-70b-versatile- LLM for content generationwhisper-large-v3- Speech-to-text transcription
- ElevenLabs - Text-to-speech (Jyot voice) for English audio cues
- Sarvam AI - Indian language TTS (Vidya voice) + Translation (mayura:v1)
- SentenceTransformers - Memory embedding (all-MiniLM-L6-v2, 384-dim)
- Qdrant Cloud - Vector similarity search
- Neo4j Cloud - Graph database
| Platform | CPU | GPU | Notes |
|---|---|---|---|
| macOS (Apple Silicon) | ✅ | ✅ CoreML | Best performance |
| macOS (Intel) | ✅ | ❌ | CPU only |
| Windows | ✅ | ✅ CUDA | Install onnxruntime-gpu |
| Linux | ✅ | ✅ CUDA | Install onnxruntime-gpu |
Purpose: Real-time face recognition and memory recording
Key State:
isRecognized: Person detected statecurrentPersonId: Matched person IDhudData: Profile data for displayisRecording: Microphone state
Hooks Used:
useFaceTracking(): 300ms recognition loopuseAudioRecorder(): Memory recording
Flow:
useEffect(() => {
if (isRecognized && currentPersonId) {
// 1. Fetch HUD data
fetchHUD();
// 2. Play whisper (if enabled)
if (whisperEnabled) {
setTimeout(() => fetchAndPlayWhisper(), 400);
}
// 3. Start recording (if enabled)
if (autoRecordEnabled) {
startRecording();
}
}
}, [isRecognized, currentPersonId]);Purpose: Enroll, edit, and manage known people
Key Features:
- Enrollment form with webcam/upload
- People grid (Bento layout)
- Edit modal
- Delete with confirmation
- Settings toggles (Memories, Whisper)
State Management:
confirmedPeople: List of enrolled peopleshowEnrollForm: Toggle enrollment UIeditingPerson: Currently editing personautoRecordEnabled,whisperEnabled: Feature toggles
Purpose: Continuous face detection and recognition
Key Logic:
// Every 300ms:
1. Capture video frame
2. Detect faces using MediaPipe
3. Send frames to /api/recognize-face
4. Update isRecognized, currentPersonId
5. If face lost for 5s → mark as idleStates:
isIdle: No face detectedisRecognized: Known person detectedisNotFound: Unknown person detected
Purpose: Display person info over video
Positioning:
- Simple offset: 80px right, 100px up from face center
- Smoothing handled by Lerp in
useFaceTracking.js - High contrast for readability
Styling:
- Name: Large, bold text
- Relation: Green pill badge (#8DA399)
- Semi-transparent white background
Key Methods:
extract_embedding(image_base64): Base64 → PIL → Face Detection → 512-dim vectorextract_embedding_from_pil(image): PIL Image → Face Detection → 512-dim vector
Model: InsightFace buffalo_s (ONNX runtime) Performance:
- Mac (CoreML): ~100-150ms per frame
- Windows (CUDA GPU): ~20-50ms per frame
- CPU fallback: ~200-300ms per frame
Device Selection: Automatically uses best available (CUDA → CoreML → CPU)
Collections:
face_embeddings(512-dim)- Payload:
person_id,status(confirmed/temporary)
- Payload:
memory_embeddings(384-dim)- Payload:
person_id,summary,emotional_tone
- Payload:
Key Methods:
search_face(embedding, limit=5): Find matching facesstore_face_embedding(person_id, embedding, status)store_memory_embedding(memory_id, person_id, embedding, ...)
Node Types:
Person: {id, name, relation, contextual_note, status, familiarity, last_seen}Memory: {id, summary, emotional_tone, important_event, raw_transcript, timestamp}Routine: {id, text, confidence, source, created_at}
Relationships:
(Person)-[:HAS_MEMORY]->(Memory)(Person)-[:HAS_ROUTINE]->(Routine)
Key Methods:
create_person(person_id, name, relation, ...)update_person(person_id, name=None, relation=None, ...)get_person(person_id)→ Returns person dictcreate_memory(person_id, summary, emotional_tone, ...)get_memories(person_id, limit=10)
Model: llama-3.3-70b-versatile via Groq API
Methods:
-
generate_hud_context(name, relation, memories, familiarity_score)- Currently returns static data (HUD is dementia-safe, no AI hallucination)
-
summarize_memory(transcript)- Input: Raw conversation text
- Output:
{summary, emotional_tone, important_event} - Temperature: 0.4 (consistency)
-
generate_whisper_text(name, relation, contextual_note, recent_memory)- Input: Person data + latest memory
- Output: 1-2 sentence calm reassurance (fallback)
-
generate_whisper(name, relation, routines, contextual_note)(NEW)- Input: Person data + database routines
- Output: 4-sentence comfort message with routines
- Structure:
- "This is [name]. He's/She's your [relation]."
- Comfort statement
- Routine/shared activity
- Reassurance ("You can take your time." / "You're safe here." / "There's no rush.")
-
extract_routines_from_memories(memories)(NEW)- Input: List of memory summaries
- Output: JSON array of extracted routine patterns
- Returns specific, concrete routines (not generic statements)
-
select_best_routine(routines, recent_memory)(NEW)- Selects most relevant routine for HUD display
System Messages:
- HUD: "You are a calm, supportive assistant helping dementia patients..."
- Memory: "You are a memory summarization assistant. Create concise, gentle summaries..."
- Whisper: "You are a calm, warm voice providing gentle reassurance..."
API: Groq Whisper API (whisper-large-v3)
Flow:
def transcribe(audio_base64):
1. Decode base64 → bytes
2. Write to temp file (Groq requires file input)
3. Call Groq API
4. Return transcript text
5. Delete temp fileAPI: ElevenLabs Text-to-Speech
Voice: Jyot (6kUBvNdOU57rLktR7BK5) - Smooth, comforting female voice
Settings: Default ElevenLabs settings (natural, clear speech)
Output: MP3 audio bytes (base64 encoded for frontend)
Purpose: Extract routine patterns from conversation memories
Key Function: analyze_and_update_routines(person_id)
- Fetches all memories for person
- Uses LLM to extract specific, concrete patterns
- Stores routines in Neo4j as
Routinenodes - Triggered by background worker every 2 memories
Purpose: Isolated background process for routine extraction
Polling: Every 30 seconds Logic:
- Query Neo4j for people needing routine analysis
- Check: memory_count % 2 == 0 AND last_routine_analysis < last_memory_saved
- Run routine extraction for each person
- Mark analysis complete
POST /api/recognize-face
Body: { images_base64: string[] }
Response: {
recognized: bool,
status: "confirmed" | "temporary",
person_id: string,
confidence: float,
message: string
}
Logic:
- Try each frame (multi-frame validation)
- Extract embedding for each
- Search Qdrant for best match
- Return only if status="confirmed" and confidence > threshold
POST /api/hud-context
Body: { person_id: string, status: string }
Response: {
name: string,
relation: string,
contextual_note: string,
speak: bool,
speechText: string
}
Logic:
- If status="temporary" → return empty (dementia-safe silence)
- If status="confirmed" → fetch from Neo4j and return static data
GET /api/whisper/{person_id}
Response: {
audio_url: string (base64 data URI),
text: string,
reason?: string
}
Logic:
- Fetch person from Neo4j
- Fetch 1 recent memory
- LLM generates script
- ElevenLabs converts to audio
- Return base64 MP3
- On any failure → silent (audio_url: null)
POST /api/memory/save
Body: { person_id: string, audio_base64: string }
Response: {
status: "saved",
memory_id: string,
summary: string,
emotional_tone: string
}
Logic:
- Groq Whisper transcribes audio
- LLM summarizes transcript
- Store in Neo4j (graph node)
- Generate embedding (SentenceTransformer)
- Store in Qdrant (vector)
POST /api/caregiver/enroll
Body: { name, relation, contextual_note, image_base64 }
Response: { person_id, status: "enrolled" }
PUT /api/caregiver/person/{person_id}
Body: { name?, relation?, contextual_note?, image_base64? }
Response: { status: "updated" }
DELETE /api/caregiver/person/{person_id}
Response: { status: "deleted", person_id }
GET /api/caregiver/confirmed
Response: { confirmed_people: Person[] }
Caregiver Input (Name, Relation, Photo)
↓
POST /caregiver/enroll
↓
FaceNet: Photo → 512-dim embedding
↓
┌─────────────┬──────────────┐
↓ ↓ ↓
Qdrant Neo4j File System
(vector) (metadata) (thumbnail)
Video Frame (300ms)
↓
POST /recognize-face
↓
FaceNet: Frame → embedding
↓
Qdrant: Vector search
↓
Match? → person_id
↓
POST /hud-context
↓
Neo4j: Fetch profile
↓
Display HUD
Recognition Event
↓
Wait 400ms (calm transition)
↓
GET /whisper/{person_id}
↓
Neo4j: Profile + Recent Memory
↓
LLM (GPT-OSS-120b): Generate script
↓
ElevenLabs: Text → Audio
↓
Play MP3 (volume 0.6)
Face Recognized → Mic ON
Face Lost → Mic OFF
↓
POST /memory/save
↓
┌──────────────┬───────────────┐
↓ ↓ ↓
Groq Whisper LLM Summary SentenceTransformer
(transcribe) (summarize) (embed)
↓ ↓ ↓
└──────────────┴───────────────┘
↓
┌─────────────┬──────────────┐
↓ ↓
Neo4j Qdrant
(graph node) (vector)
- Captures 3 frames
- Processes each independently
- Returns best match
- Confidence threshold: 0.6
- Only returns "confirmed" status people
- Static Data: No AI generation (zero hallucination risk)
- High Contrast: Black text on semi-opaque white background
- Positioned: Offset from face to avoid overlap
- Persistent: Stays visible while face is detected
- One-Time: Plays once per session
- Delayed: 400ms after recognition (calm transition)
- LLM-Generated: Personalized using profile + recent memory
- Toggleable: Caregiver can disable
- Silent Failure: No audio if any step fails
- Automatic: Starts on recognition (if enabled)
- Hands-Free: No button press needed
- Smart Stop: Stops when face leaves frame
- Summarized: LLM extracts key points
- Searchable: Vector embedding for future semantic search
- Edit in Place: Modal overlay for quick updates
- Photo Update: Can replace face photo (re-generates embedding)
- Immediate Sync: Updates both Neo4j and Qdrant
- Confirmation: Delete requires confirmation dialog
- Calm Over Cleverness: Predictable, not surprising
- Stability Over Responsiveness: Consistent UI, no sudden changes
- Familiarity Over Novelty: High-contrast text, simple layouts
- Visual First: HUD is static and instant
- Audio Second: Whisper is gentle and optional
- Caregiver Authority: Only caregivers can enroll/edit
- Toggle Controls: Caregivers control recording and whisper
- Local Storage: Preferences persist on device
- No Cloud Storage: Face images stored locally on server
- Real-time Recognition: 300ms loop
- Immediate HUD: <100ms display
- Calm Audio: 400ms delay (intentional)
- Background Processing: Memory save non-blocking
Frontend (.env):
VITE_API_URL=http://localhost:8000/api
Backend (.env):
# Groq API
GROQ_API_KEY=gsk_...
# Qdrant Cloud
QDRANT_URL=https://...
QDRANT_API_KEY=...
# Neo4j Cloud
NEO4J_URI=neo4j+s://...
NEO4J_USER=neo4j
NEO4J_PASSWORD=...
# ElevenLabs
ELEVENLABS_API_KEY=sk_...
ELEVENLABS_VOICE_ID=21m00Tcm4TlvDq8ikWAM
# Config
FACE_SIMILARITY_THRESHOLD=0.8
Frontend:
cd frontend
npm install
npm run devBackend:
cd backend
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -m uvicorn app.main:app --reloadexport async function recognizeFace(imagesBase64) {
const response = await fetch(`${API_URL}/recognize-face`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ images_base64: imagesBase64 }),
});
if (!response.ok) throw new Error('Recognition failed');
return response.json();
}export function useFaceTracking(videoRef, onRecognize) {
const [isRecognized, setIsRecognized] = useState(false);
const [currentPersonId, setCurrentPersonId] = useState(null);
useEffect(() => {
// 300ms recognition loop
const interval = setInterval(async () => {
const frames = captureFrames(videoRef.current);
const result = await recognizeFace(frames);
if (result.recognized) {
setIsRecognized(true);
setCurrentPersonId(result.person_id);
onRecognize(result);
}
}, 300);
return () => clearInterval(interval);
}, []);
return { isRecognized, currentPersonId, ... };
}const [whisperEnabled, setWhisperEnabled] = useState(() => {
return localStorage.getItem('cue_whisperEnabled') !== 'false';
});
const toggleWhisper = () => {
const newValue = !whisperEnabled;
setWhisperEnabled(newValue);
localStorage.setItem('cue_whisperEnabled', String(newValue));
};class FaceRecognitionService:
def __init__(self):
self.model = None
self._initialized = False
def initialize(self):
if self._initialized:
return
self.model = InceptionResnetV1(pretrained='vggface2').eval()
self._initialized = True
# Singleton
face_recognition = FaceRecognitionService()router = APIRouter(tags=["Memory"])
@router.post("/memory/save", response_model=MemorySaveResponse)
async def save_memory(request: MemorySaveRequest):
# Business logic
transcript = whisper_service.transcribe(request.audio_base64)
memory_data = llm_service.summarize_memory(transcript)
memory_id = graph_db.create_memory(...)
return MemorySaveResponse(...)try:
audio_bytes = await generate_speech(whisper_text)
if not audio_bytes:
return WhisperResponse(reason="tts_failed")
except Exception as e:
print(f"⚠️ Whisper error: {e}")
return WhisperResponse(reason="generation_failed")- Update Task List:
brain/task.md - Backend:
- Add endpoint to
app/routers/ - Add service to
app/services/if needed - Update schemas in
app/models/schemas.py - Register router in
app/main.py
- Add endpoint to
- Frontend:
- Add API function to
services/api.js - Update component (PatientMode or CaregiverMode)
- Add CSS to corresponding
.cssfile
- Add API function to
-
Check Console Logs:
- Frontend: "🔍 Processing X frames..."
- Backend: "Frame 1: Matched ... (confidence: 0.XX)"
-
Verify Database:
- Qdrant: Check collection size
- Neo4j: Query Person nodes
-
Test Embedding:
- Use
/api/recognize-facewith test image - Check confidence scores
- Use
Edit backend/app/services/llm.py:
- Adjust
temperature(lower = more consistent) - Modify system message
- Change prompt structure
- Update
max_tokens
Cause: Import order issue
Fix: Ensure from pydantic import BaseModel is at top of file
Causes:
- Webcam not accessible
- Face too small/blurry
- Person not enrolled
- Qdrant not connected
Debug:
# Check Qdrant connection
curl http://localhost:8000/api/health
# Check enrollment
curl http://localhost:8000/api/caregiver/confirmedCauses:
- Toggle disabled
- ElevenLabs API key missing
- Browser autoplay policy
Check:
localStorage.getItem('cue_whisperEnabled')- Backend logs for ElevenLabs errors
- Browser console for audio errors
Causes:
- Groq API key missing
- Audio recording failed
- Neo4j not connected
Debug: Check backend logs for transcription/summarization errors
- Enroll person with webcam
- Enroll person with file upload
- Edit person name/relation/note
- Edit person photo (verify re-embedding)
- Delete person (verify from both DBs)
- Toggle "Memories" on/off
- Toggle "Whisper" on/off
- Face recognized within 2 seconds
- HUD displays correct name/relation
- Whisper plays after 400ms (if enabled)
- Recording starts on recognition (if enabled)
- Recording stops when face leaves
- Memory saved and appears in Neo4j
frontend/src/pages/PatientMode.jsx- Main user experiencebackend/app/main.py- API structurebackend/app/services/face_recognition.py- Face matchingbackend/app/services/llm.py- Content generationfrontend/src/hooks/useFaceTracking.js- Recognition loop
| Component | File Path |
|---|---|
| Recognition Loop | frontend/src/hooks/useFaceTracking.js |
| Visual HUD | frontend/src/components/HUD.jsx |
| Memory Recording | frontend/src/hooks/useAudioRecorder.js |
| Face API | backend/app/routers/recognize.py |
| Whisper API | backend/app/routers/whisper.py |
| Memory API | backend/app/routers/memory.py |
| Caregiver API | backend/app/routers/caregiver.py |
| FaceNet Service | backend/app/services/face_recognition.py |
| Qdrant Service | backend/app/services/vector_db.py |
| Neo4j Service | backend/app/services/graph_db.py |
| LLM Service | backend/app/services/llm.py |
| STT Service | backend/app/services/whisper.py |
| TTS Service | backend/app/services/elevenlabs.py |
This document should give you (Claude or any LLM) a complete understanding of the Cue codebase. For specific implementation details, refer to the actual code files mentioned above.