A real‑time, embodied, multi‑agent voice assistant for the NAO H25 humanoid. Built at Morgan State University · Department of Computer Science.
Aayush Shrestha · Advised by Dr. Shuangbao "Paul" Wang · Department of Computer Science, Morgan State University
A NAO humanoid that listens, sees, thinks, talks, and moves like a person. A student walks up, NAO recognizes their face, greets them by name, answers questions about courses, weather, or feelings, gestures while talking, turns its head toward whoever speaks, and runs a hardcoded crisis gate that bypasses the LLM entirely on suicidal ideation.
End‑to‑end target: < 800 ms p50 mouth‑close to first audio.
For people new to this repo: read this README, then
docs/DECISIONS(how problems were navigated), thendocs/PRD_v2(full spec). Index of all docs indocs/INDEX.
sequenceDiagram
autonumber
participant U as User
participant N as NAO
participant S as Server
participant A as Agents SDK
participant T as ElevenLabs
U->>N: speech
N->>S: WS audio_chunk frames
S->>S: VAD plus Silero gate
S->>S: STT (Deepgram or Whisper)
S-->>S: Crisis gate (988 hard)
S-->>S: Motion trigger short-circuit
S->>A: Router to specialist
A-->>S: streaming token deltas
S->>S: Sentence chunker
S->>T: per-sentence synth (parallel)
T-->>S: MP3 chunks
S->>N: WS audio_chunk plus action frames
N->>N: ALAudioPlayer plus gestures
N-->>U: speech plus motion
N->>N: post-playback drain
N->>S: mic_resumed (clean restart)
graph LR
classDef u fill:#0d47a1,stroke:#0d47a1,color:#fff
classDef nao fill:#1565c0,stroke:#1565c0,color:#fff
classDef srv fill:#2e7d32,stroke:#2e7d32,color:#fff
classDef ai fill:#412991,stroke:#412991,color:#fff
classDef ext fill:#ef6c00,stroke:#ef6c00,color:#fff
classDef gate fill:#b71c1c,stroke:#b71c1c,color:#fff
classDef db fill:#6a1b9a,stroke:#6a1b9a,color:#fff
USER([User]):::u
subgraph NAO["NAO H25 naoqi 2.7"]
WAKE["Wake state machine"]:::nao
WSC["WS client"]:::nao
AUD["Audio module<br/>fragment recorder"]:::nao
TTS["Stream TTS<br/>MP3 to WAV to ALAudioPlayer"]:::nao
EXEC["Action worker<br/>gestures, dances, motors"]:::nao
SND["Sound localizer<br/>plus ALTracker face mode"]:::nao
BRAIN[("Brain cache<br/>64 KB")]:::db
end
subgraph SRV["Server FastAPI plus uvicorn"]
WSS["WebSocket ws endpoint"]:::srv
SAFE["Crisis gate"]:::gate
MTR["Motion trigger"]:::srv
VAD["Silero VAD"]:::srv
STT["STT lane<br/>Deepgram, Whisper, Scribe"]:::srv
ROUTER["Router"]:::ai
CHAT["Chat lanes<br/>pure and embodied"]:::ai
CHATBOT["Chatbot"]:::ai
SKILLS["Skills"]:::ai
THER["Therapist plus CBT, Grounding, MI"]:::ai
VIS["GPT-4o vision<br/>lazy, fresh per question"]:::ai
SES[("SQLite session<br/>plus user_prefs")]:::db
OBS["metrics plus JSONL"]:::srv
end
EL[("ElevenLabs Flash v2.5")]:::ext
OAI[("OpenAI Agents SDK")]:::ext
CSN[("CS Navigator API<br/>Cloud Run")]:::ext
USER -->|voice| WAKE --> WSC
USER -->|face/proximity| WAKE
AUD --> WSC -->|audio_chunk| WSS
WSS --> VAD --> STT
STT --> SAFE
SAFE -.crisis.-> WSS
SAFE --> MTR
MTR -.short-circuit.-> WSS
MTR --> ROUTER
ROUTER --> CHAT & CHATBOT & SKILLS & THER
CHATBOT --> CSN
THER -.vision.-> VIS
CHAT & CHATBOT & SKILLS & THER --> OAI
OAI -->|stream tokens| WSS
WSS --> EL
EL -->|MP3| WSS
WSS -->|audio_chunk + action| WSC
WSC --> TTS --> EXEC
SND --> EXEC
BRAIN <-->|sync| WSS
SES -.persistence.-> ROUTER
OBS -.observability.-> WSS
graph TD
classDef gate fill:#b71c1c,stroke:#b71c1c,color:#fff
classDef route fill:#0d47a1,stroke:#0d47a1,color:#fff
classDef spec fill:#1565c0,stroke:#1565c0,color:#fff
classDef sub fill:#7b1fa2,stroke:#7b1fa2,color:#fff
classDef tool fill:#2e7d32,stroke:#2e7d32,color:#fff
T([turn]) --> CG{crisis check}:::gate
CG -- positive --> H["988 hotline<br/>fixed reply, no LLM"]:::gate
CG -- clean --> MT{motion trigger}:::route
MT -- match --> ACT["short-circuit action"]:::route
MT -- no match --> R["router<br/>gpt-4.1-nano"]:::route
R --> CH["chat<br/>pure and embodied"]:::spec
R --> CB["chatbot"]:::spec
R --> SK["skills"]:::spec
R --> TH["therapist<br/>gpt-4.1-mini"]:::spec
TH --> CBT["cbt_coach<br/>thought records"]:::sub
TH --> GR["grounding_coach<br/>5-4-3-2-1, box, scan"]:::sub
TH -. opt .-> MI["mi_coach<br/>OARS"]:::sub
CH & TH -.-> NA[("nao_actions")]:::tool
CB -.-> CSN[("cs_navigator")]:::tool
SK -.-> ST[("skills_tools")]:::tool
TH & CBT & GR -.-> EM[("emotion plus memory")]:::tool
CH -.-> LF[("learn_face")]:::tool
| Agent | Role | Default model |
|---|---|---|
| router | triage + handoff (sensory grounding rule, never denies senses) | gpt-4.1-nano |
| chat (pure) | tool‑less ultra‑fast lane, < 2 s first audio | gpt-4.1-nano |
| chat (embodied) | gestures + actions + face learn | gpt-4.1-nano |
| chatbot | Morgan‑CS RAG via CS Navigator API | gpt-4.1-mini |
| skills | time, weather, timers, todos | gpt-4.1-nano |
| therapist | empathy + handoffs + vision | gpt-4.1-mini |
| cbt_coach | Beck thought record (one step per turn) | gpt-4.1-mini |
| grounding_coach | 5‑4‑3‑2‑1, box breathing, body scan | gpt-4.1-mini |
| mi_coach | Motivational Interviewing (OARS) — experimental | gpt-4.1-mini |
| crisis | safety classifier (soft triggers only) | gpt-4.1 |
stateDiagram-v2
direction LR
[*] --> IDLE
IDLE --> AWARE: face detected (conf >= 0.35)
AWARE --> IDLE: face lost or 8 s timeout
AWARE --> ENGAGED: mutual gaze 1.5s, proximity, speech, or hey NAO
ENGAGED --> LISTENING: chime + WS open
LISTENING --> SPEAKING: TTS playing
SPEAKING --> LISTENING: playback drained + grace
LISTENING --> AWARE: face lost > 5 s
SPEAKING --> LISTENING: barge-in (touch / speech onset)
note right of AWARE
Robot is aware but silent.
No greeting, no chime —
prevents passerby false-wakes.
end note
note right of LISTENING
Eyes cyan - gaze drift
Mic open - streaming PCM
end note
flowchart TD
classDef a fill:#0d47a1,stroke:#0d47a1,color:#fff
classDef ok fill:#2e7d32,stroke:#2e7d32,color:#fff
classDef bad fill:#b71c1c,stroke:#b71c1c,color:#fff
START([reply ready]) --> ENQ["stream_tts.enqueue<br/>per-sentence MP3"]:::a
ENQ --> PLAY["ALAudioPlayer<br/>blocking_play_start"]:::a
PLAY --> DONE{"queue empty?"}:::a
DONE -- no --> NEXT["next chunk"]:::a --> PLAY
DONE -- yes --> DRAIN["local_tts_queue_empty"]:::ok
DRAIN --> SETTLE["wait 800 ms<br/>speaker cone settle"]:::a
SETTLE --> OPEN["gate False<br/>fresh stream.wav"]:::ok
OPEN --> RESUME["push mic_resumed"]:::ok
RESUME --> LISTEN([mic live])
ECHO{"server self_echo<br/>reject?"}:::bad
LISTEN --> ECHO
ECHO -- yes --> CLOSE["gate True<br/>stop recorder"]:::bad
CLOSE --> RESETTLE["250 ms settle"]:::bad
RESETTLE --> RESTART["gate False<br/>fresh stream.wav"]:::ok
RESTART --> LISTEN
The old design opened the mic on the server's tts_ended frame — but that only means the server stopped sending audio. The robot's local queue could still play for 5–8 more seconds, and the mic would record NAO's own speaker output. See DECISIONS § D8.
sequenceDiagram
participant Server
participant Recv as WS recv loop
participant Q as Action queue
participant Worker as Action worker
participant NAO as ALMotion plus ALBehavior
Server->>Recv: action frame
Recv->>Q: put_nowait
Recv-->>Server: returns immediately
Worker->>Q: get
Worker->>NAO: dispatch, blocking up to 15 s
NAO-->>Worker: done
Worker->>Q: get next
Note over Server,NAO: barge-in or crisis or shutdown
Server->>Recv: control barge_in
Recv->>Worker: drain queue
Recv->>NAO: stopAllBehaviors
NAO-->>Worker: cancelled
See DECISIONS § D9, § D10.
sequenceDiagram
participant U as User
participant W as Wake SM
participant N as NAO
participant S as Server
participant A as Chat agent
U->>W: walks into camera view
W->>W: AWARE to ENGAGED, gate fires
N->>S: control session_open
N->>S: image (reason=session_open)
N-->>U: "Heads up - my camera is on..."
par face scan (3 s daemon)
N->>N: ALFaceDetection.recognize
alt known face
N->>S: user_identified, name Aayush, recognized true
else unknown
N->>S: user_identified, name null, face_visible true
end
end
U->>S: "hey can you see me"
S->>A: USER name=Aayush returning=true + NAO_VISION + transcript
alt known
A-->>U: "Welcome back, Aayush!"
else unknown
A-->>U: "I see you. What's your name?"
U->>S: "I'm Aayush, remember me"
S->>A: route to chat
A->>N: tool: learn_face(name=Aayush)
N->>N: ALFaceDetection.learnFace
A-->>U: "Got it, Aayush. Pleasure to meet you."
end
|
|
flowchart LR
classDef step fill:#0d47a1,stroke:#0d47a1,color:#fff
A["clone repo"]:::step --> B["python -m venv .venv<br/>pip install -r server/requirements.txt"]:::step
B --> C["cp .env.example .env<br/>fill OPENAI_API_KEY etc."]:::step
C --> D["./run.sh"]:::step
D --> E(["uvicorn on 5050<br/>plus NAO main.py"])
git clone https://github.com/theaayushstha1/nao-sagecbt.git
cd nao-sagecbt
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r server/requirements.txt
cp .env.example .env # fill keys
./run.sh # deploy to robot + boot uvicorn + tail logs
./run.sh stop # tear downrun.sh rsyncs nao/ to /home/nao/nao_assist/, boots uvicorn server.app_ws:app on :5050, launches main.py on the robot, and tails a signal‑filtered combined log (set RAW_LOGS=1 to see everything).
Dev mode runs the server on your laptop via ./run.sh. For demos and 24/7 availability we now also host the server on a Raspberry Pi 4 (Ubuntu Server 24.04) so the robot is reachable without anyone's laptop being online.
flowchart LR
classDef box fill:#0d47a1,stroke:#0d47a1,color:#fff
NAO["NAO robot<br/>172.20.95.127<br/>Choregraphe autostart<br/>→ launch_nao_assist.sh"]:::box
PI["Raspberry Pi 4<br/>172.20.95.106<br/>systemd nao-server.service<br/>uvicorn server.app_ws:app"]:::box
NAO -- "WebSocket :5050<br/>audio_chunk / control / image" --> PI
PI -- "audio_chunk MP3 / action / transcript" --> NAO
- Robot side: power on → NAOqi boots → Choregraphe default behavior
nao-therapy-autostart/behavior_1fires →/home/nao/launch_nao_assist.shmutes NAO's native ALTextToSpeech (ElevenLabs is the only voice), disables autonomous life, and launchespython /home/nao/nao_assist/main.py. Nothing to do by hand. - Server side: Pi runs
nao-server.service(enabled at boot). Server pulls latestmainongit pull && sudo systemctl restart nao-server.
# Deploy code updates to the Pi server
ssh -t nao@172.20.95.106 'cd ~/nao-sagecbt && git pull && sudo systemctl restart nao-server && sudo systemctl is-active nao-server'
# Deploy code updates to the robot
./run.sh deploy-only # rsyncs full nao/ folder + clears .pyc
# Restart main.py on the robot
ssh -t nao@172.20.95.127 'pkill -9 -f main.py; sleep 2; bash /home/nao/launch_nao_assist.sh'
# Tail server logs
ssh -t nao@172.20.95.106 'sudo journalctl -u nao-server -f'The robot's ~/.bash_profile previously auto‑started a separate Feb‑2026 project that collided with the new app over the mic and speaker (the "two main.py at different paths" bug); that autostart is now neutralized — see ~/.bash_profile.old_nao_chat_backup.* for the original.
- Pi migration: server now runs on a Raspberry Pi 4 via systemd; robot autostarts via Choregraphe default behavior. The robot can be unboxed, powered on, and talked to without any laptop.
- Mic‑stall watchdog (
nao/audio_module.py): detects whenALAudioRecorderwedges (WAV stops growing) and restarts the recorder after 2 s. Prevents mid‑conversation silence. - VAD adaptive threshold (
server/vad_silero.py): Silero VAD threshold lowered to 0.15 with an adaptive floor at 0.05 so NAO actually hears quiet voices instead of rejecting every turn asno_voice. Per‑second[silero_trace]log line for tuning. - Durable
had_speech(server/app_ws.py,server/vad_silero.py): per‑utterance flag captures speech bursts that end on a quiet syllable, so the end‑of‑utterance arbiter doesn't drop the turn. - TTS sentence dedupe (
server/app_ws.py): prevents the agent from speaking the same sentence twice when an LLM tool call interrupts streaming. - Native voice locked off (
nao/main.py,nao/stream_tts.py,nao/ws_client.py):ALTextToSpeech.setVolume(0.0)is now pinned at every collision point (boot, TTS playback, announcer fillers gated behindENABLE_NATIVE_FILLER=1). NAO's kid voice never leaks. learn_facefast‑path (server/motion_trigger.py): "remember me as X" / "save my face as X" / "my name is X" short‑circuit before any agent routes, so face learning works from any mode (router, chat, therapy, morgan).- Persistent therapy memory (
server/session.py,server/tools/emotion.py,server/memory.py): two new SQLite tables —mood_logandthought_records— populated by thelog_emotion,identify_distortion, andsuggest_reframetools. The system preamble now surfaces aTherapy memorysection with the latest mood, a 5‑entry trajectory, and the last CBT thought record so the therapist can open the next session with continuity. - Proactive name use (
server/agents/therapist.py,server/agents/chat.py): both agents weave the user's name into ~1 in 3 replies (greetings, transitions, validations, peaks). Memory‑aware first turn for the therapist references recent mood / thought records.
| Say | Triggers |
|---|---|
| "Hey NAO" / step into view | Wake state machine |
| "Wave at my friend" | wave_hand action |
| "Do the kung fu" | dance(style='kungfu') -> KungFu_1 Choregraphe pack |
| "Follow me" / "Track me" | follow_movement (follow-me pack) |
| "Stop following me" / "Freeze" | stop_follow |
| "What am I wearing?" / "Can you see me?" | Lazy GPT‑4o vision call |
| "Switch to a man voice" / "Use my voice" | Voice profile flip (per‑user persisted) |
| "Remember me as Aayush" | learn_face(name='Aayush') |
| "Stop watching me" | Camera off for session |
| "Set a 10‑minute timer" | Skills agent |
| "I'm anxious about finals" | Therapist agent |
| "What classes does Morgan offer in spring?" | CS Navigator (chatbot agent) |
Tap NAO's head sensors at any time to barge in — TTS stops within ~200 ms, current behavior cancels via stopAllBehaviors().
| Metric | Target | Where measured |
|---|---|---|
e2e_user_to_first_audio p50 |
< 800 ms | phase_ms in turn_complete log |
e2e_user_to_first_audio p95 |
< 1.2 s | Prometheus histogram |
tts_synth_first_chunk |
< 500 ms | Flash v2.5 streaming |
vision_call |
~1.5 s | only on visual triggers |
| Barge‑in stop time | < 200 ms | tts_player.stop() + stopAllBehaviors |
/metrics exposes histograms per phase. Grafana JSON in server/dashboards/grafana_voice.json.
nao-sagecbt/
├── nao/ Python 2.7 — runs on the robot
│ ├── main.py entry: wake SM + session controller
│ ├── ws_client.py long-lived WebSocket session
│ ├── audio_module.py ALAudioRecorder fragment streamer
│ ├── stream_tts.py MP3 -> WAV -> ALAudioPlayer + ffmpeg loudness
│ ├── wake_state.py IDLE -> AWARE -> ENGAGED -> LISTENING -> SPEAKING
│ ├── sound_localize.py ALSoundLocalization auto-track
│ ├── leds.py eye LED helpers
│ ├── idle_motion.py background breathing + gaze drift
│ └── utils/
│ ├── nao_execute.py dispatcher: gestures, dances, motors
│ ├── face_naoqi.py face recognition + learning
│ ├── brain.py 64 KB local identity cache
│ └── camera_capture.py per-turn JPEG snap
│
├── server/ Python 3.11+ — runs on dev / cloud
│ ├── app_ws.py FastAPI + WebSocket — main entry
│ ├── safety.py pre-dispatch crisis gate (988)
│ ├── motion_trigger.py regex short-circuit for body commands
│ ├── streaming.py sentence chunker for streaming TTS
│ ├── elevenlabs_tts.py Flash v2.5 streaming with voice profiles
│ ├── deepgram_asr.py Nova-2 streaming STT
│ ├── elevenlabs_stt.py Scribe v2 Realtime STT
│ ├── vad_silero.py server-side authoritative voice gate
│ ├── memory.py recaps, weekly themes, monthly personas
│ ├── session.py SQLiteSession + camera consent
│ ├── logging_setup.py structlog JSON
│ ├── metrics.py Prometheus exporter
│ ├── dashboards/ Grafana JSON
│ ├── agents/ router, chat, chatbot, skills, therapist, ...
│ └── tools/ nao_actions, cs_navigator, emotion, skills
│
├── sim/ Python 3.11+ — load test + benchmarks
├── docs/ PRD, DECISIONS, INDEX, phase task maps
└── run.sh one-shot deploy + boot + log tail
| Tier | Doc | What it gives you |
|---|---|---|
| 1 | docs/INDEX |
Doc graph + tag glossary |
| 1 | docs/DECISIONS |
How problems were navigated — 12 key decisions with the why |
| 1 | docs/PRD_v2 |
600‑line product/architecture spec, 9 phases |
| 2 | docs/spike_results |
Phase 0.5 — FastAPI WS vs Realtime API benchmark |
| 3 | docs/PHASE_*_TASK_MAP |
Per‑phase delivery plans with shared contracts |
Every doc carries an HTML‑comment frontmatter block with tags: and related: so they're greppable and graph‑traversable. Search example:
grep -lZ "tags:.*embodiment" docs/*.md # every doc tagged with embodiment
grep -lZ "related:.*PHASE_4" docs/*.md # every doc that links to Phase 4.env essentials (full list in .env.example, never commit .env):
OPENAI_API_KEY=sk-...
ELEVENLABS_API_KEY=...
ELEVENLABS_VOICE_GIRL=21m00Tcm4TlvDq8ikWAM
ELEVENLABS_VOICE_MAN=...
ELEVENLABS_VOICE_NEUTRAL=...
ELEVENLABS_VOICE_MY=... # cloned "my voice" (optional)
DEEPGRAM_API_KEY=... # optional, fastest STT
NAO_IP=172.20.95.127 # robot LAN IP
NAO_PASSWORD=... # SSH for run.sh deploy (gitignored)
NAO_SHARED_SECRET=... # WS auth between robot and server
SERVER_IP=192.168.x.x # this machine's LAN IP
USE_WS=1 # FastAPI + WebSocket transportFull env reference: see server/config.py.
pytest -q # 30 unit + integration tests
python -m sim.live_proof # full pipeline soak test
python -m sim.stt_ab # A/B Deepgram vs Whisper vs Scribe
python -m sim.chat_model_bench # per-model latencyssh nao@$NAO_IP # CS network
rsync -avz --delete nao/ nao@$NAO_IP:/home/nao/nao_assist/
qicli call ALBehaviorManager.getInstalledBehaviors # 915 behaviors
qicli call ALBehaviorManager.runBehavior "follow-me"
qicli call ALBehaviorManager.stopAllBehaviorsVS Code Remote‑SSH:
Host nao
HostName 172.20.95.127
User nao
- Dr. Shuangbao "Paul" Wang for advising the SAGE‑CBT research direction
- Morgan State Department of Computer Science for NAO + lab access
- OpenAI, Anthropic Claude Code, ElevenLabs, Deepgram, Aldebaran/SoftBank Robotics
MIT — see LICENSE.