CLAUDE.md - Agentic System Guide

Last Updated: 2025-12-07 For: Future Claude instances working on this codebase

🚀 Quick Start

New to this project? Start here:

Quick Start Guide - Get running in 5 minutes
Operational Context - Understand your environment (Local Dev vs Production)
Documentation Index - Complete documentation map

Essential Context
Project Overview
Architecture
Development Guides
System Components
Deployment
Quick Reference

Essential Context

Detect Your Environment First

hostname && pwd

Two operational contexts exist:

Local Development (VS Code) - Direct code editing, localhost URLs
Jetson Production - Voice assistant, systemd services, HTTPS URLs

📖 See: OPERATIONAL_CONTEXT.md for complete context guide.

⚠️ CRITICAL: Node.js Environment Differences

Local Development (uses NVM):

~/.nvm/versions/node/v22.21.1/bin/npm start

Jetson Production (uses Conda):

export PATH=/home/rodrigo/miniconda3/envs/agentic/bin:$PATH
npm run build

Note: Never use bare npm or node commands - always use full paths or set PATH first.

Project Overview

An agentic AI system with Python backend (FastAPI + AutoGen) and React frontend.

Key Features

Multi-agent coordination - Nested team agents with orchestration
Multimodal vision agents - Interpret images from tool responses
Voice assistant - OpenAI Realtime API with WebRTC bridge
Mobile voice interface - Smartphone as wireless microphone
Claude Code integration - Live code self-editing
Memory management - ChromaDB + MongoDB storage
Screenshot workflows - Automated UI development

Tech Stack

Backend:

Python 3.x, FastAPI, AutoGen
ChromaDB 0.4.24, MongoDB 3.6+, SQLite
OpenAI API, Anthropic API

Frontend:

React 18, Material-UI
WebSocket + WebRTC
Feature-based architecture

Architecture

Directory Structure

/home/rodrigo/agentic/
├── backend/                    # Python FastAPI server
│   ├── main.py                 # FastAPI entry point
│   ├── agents/                 # Agent JSON configurations
│   ├── tools/                  # Custom tool implementations
│   ├── core/                   # Agent execution engine
│   ├── api/                    # Voice & Claude Code controllers
│   ├── config/                 # Configuration & schemas
│   ├── utils/                  # Utility modules
│   └── tests/                  # Test suite
│
├── frontend/                   # React application
│   └── src/features/           # Feature-based organization
│       ├── agents/             # Agent management
│       ├── tools/              # Tool management
│       └── voice/              # Voice assistant
│
├── debug/                      # Screenshot & export tools
├── docs/                       # Documentation
└── CLAUDE.md                   # This file

Data Flow

User Input → WebSocket → FastAPI → Agent Factory → Agent Execution
    → Tool Execution → WebSocket Stream → Frontend → UI Display

Important URLs

Local Development:

Backend: http://localhost:8000
Frontend: http://localhost:3000/agentic/

Production (Jetson):

Backend: https://192.168.0.200/api/
Frontend: https://192.168.0.200/agentic/

Development Guides

Agent Development

Agent Types:

looping - Single agent with tool loop
nested_team - Multi-agent coordination
multimodal_tools_looping - Vision-capable agent
dynamic_init_looping - Custom initialization logic

Create New Agent:

Create JSON in backend/agents/YourAgent.json
Configure tools, LLM, prompts
Test via frontend

📖 Complete Guide: Creating New Agents (see full CLAUDE.md)

Tool Development

Tool Structure:

# tools/my_tool.py
from autogen_core.tools import FunctionTool

def my_tool(param: str) -> str:
    """Tool description for LLM"""
    return f"Result: {param}"

my_tool_func = FunctionTool(
    func=my_tool,
    name="my_tool",
    description="Brief description"
)

tools = [my_tool_func]

📖 Complete Guide: Creating New Tools (see full CLAUDE.md)

Voice System

WebRTC Bridge Architecture:

Browser (WebRTC) ↔ Backend (aiortc) ↔ OpenAI Realtime API (WebRTC)

Start Interactive Session:

# Terminal 1
./start-backend.sh

# Terminal 2
./start-frontend.sh

# Browser: http://localhost:3000/agentic/voice

📖 Complete Guides:

Voice Quick Start - 1-minute setup
Voice Interactive Guide - Full walkthrough
Voice System Overview - Testing guide

Mobile Voice Interface

Purpose: Use smartphone as wireless microphone

Access: http://[YOUR_IP]:3000/agentic/mobile-voice

📖 Complete Guide: docs/guides/MOBILE_VOICE_GUIDE.md

System Components

Database & Memory

Two Systems:

MongoDB - Structured data (Database agent, 10 tools)
ChromaDB - Vector memory (Memory agent, 8 tools)

📖 Complete Setup: docs/guides/DATABASE_AND_MEMORY_SETUP.md

Debugging Tools

Screenshot Automation:

~/.nvm/versions/node/v22.21.1/bin/node debug/screenshot.js http://localhost:3000/agentic/voice

Voice Conversation Export:

python3 debug/export_voice_conversations.py

📖 Complete Guide: Debugging Tools & Workflows (see full CLAUDE.md)

Claude Code Integration

Self-editing capabilities via voice:

Voice model calls: send_to_claude_code({text: "Add feature"})
Claude Code executes with bypassPermissions mode
Events stream back via WebSocket

📖 Complete Guide: Claude Code Self-Editor (see full CLAUDE.md)

Deployment

Jetson Nano Production

Server: 192.168.0.200 (ARM64, Ubuntu 18.04.6 LTS)

Quick Access:

# SSH
ssh rodrigo@192.168.0.200

# HTTPS
https://192.168.0.200/agentic/

Environment: Miniconda3 with agentic conda environment (Python 3.11, Node 20.17)

Common Tasks:

# Set PATH for conda environment (required for non-interactive SSH)
export PATH=/home/rodrigo/miniconda3/envs/agentic/bin:$PATH

# Deploy frontend update
cd ~/agentic/frontend
npm install  # if new dependencies
npm run build
sudo kill -HUP $(cat ~/nginx.pid)

# Restart backend
sudo systemctl restart agentic-backend

# View logs
sudo journalctl -u agentic-backend -f

📖 Complete Guide: docs/deployment/JETSON_DEPLOYMENT_GUIDE.md

Quick Reference

Common Commands

# Start services (local dev)
cd backend && source venv/bin/activate && uvicorn main:app --reload
cd frontend && ~/.nvm/versions/node/v22.21.1/bin/npm start

# Run tests
cd backend && pytest tests/ -v

# Take screenshot
~/.nvm/versions/node/v22.21.1/bin/node debug/screenshot.js http://localhost:3000/agentic/voice

# Export voice data
python3 debug/export_voice_conversations.py

# List agents/tools
curl http://localhost:8000/api/agents
curl http://localhost:8000/api/tools

WebRTC Voice Commands

# Test WebRTC (no network)
pytest tests/integration/test_backend_webrtc_integration.py -v

# Check active sessions
curl http://localhost:8000/api/realtime/conversations

# Stop voice bridge
curl -X DELETE http://localhost:8000/api/realtime/webrtc/bridge/{conversation_id}

# Monitor logs
tail -f logs/backend.log | grep -E "(webrtc|openai)"

File Locations

Purpose	Location
Agent configs	`backend/agents/*.json`
Tool implementations	`backend/tools/*.py`
Voice controller	`backend/api/realtime_voice_webrtc.py`
OpenAI WebRTC client	`backend/api/openai_webrtc_client.py`
Frontend voice	`frontend/src/features/voice/pages/`
Screenshot tool	`debug/screenshot.js`
Voice DB exports	`debug/db_exports/voice_conversations/`
Documentation	`docs/`

Best Practices

Development Workflow

Always Use TodoWrite - Track multi-step tasks
Read Before Write - Understand context before changes
Test Changes - Screenshots for UI, exports for backend
Screenshot Before/After - Visual verification for UI changes

UI Development

# 1. Screenshot before
node debug/screenshot.js http://localhost:3000/agentic/voice before.png

# 2. Make changes

# 3. Wait for hot reload
sleep 3

# 4. Screenshot after
node debug/screenshot.js http://localhost:3000/agentic/voice after.png

# 5. Read and verify

Agent Development

Design agent purpose - What specific task?
Choose agent type - Looping or nested team?
Select tools - What capabilities needed?
Write system prompt - Clear instructions + examples
Test iteratively - Start simple, add complexity

Troubleshooting

Backend Issues

# Port conflict
lsof -i :8000

# Dependencies
cd backend && pip install -r requirements.txt

# MongoDB not running
sudo systemctl start mongodb

Frontend Issues

# Always use nvm path!
export NODE_PATH=~/.nvm/versions/node/v22.21.1/bin

# Port conflict
lsof -i :3000

# Reinstall dependencies
cd frontend && $NODE_PATH/npm install

WebRTC Issues

# Check browser console for ICE state
# Expected: new → checking → connected → completed

# Backend logs
tail -f logs/backend.log | grep -i webrtc

# Verify OPENAI_API_KEY
grep OPENAI_API_KEY backend/.env

Documentation Index

Core Documentation

QUICK_START.md - Get started in 5 minutes
OPERATIONAL_CONTEXT.md - Context-specific behavior
DOCUMENTATION_INDEX.md - Complete doc map

Feature Guides

DATABASE_AND_MEMORY_SETUP.md - MongoDB & ChromaDB
MULTIMODAL_AGENT_GUIDE.md - Vision agents
MOBILE_VOICE_GUIDE.md - Smartphone microphone

Voice System

Voice System Overview - Architecture & integration
Voice Quick Start - 5-minute setup
Voice Interactive Guide - Complete walkthrough
Voice Commands - Command reference
Voice Troubleshooting - Debug guide

Voice Technical Details

Backend Implementation - WebRTC bridge (coming soon)
Frontend Implementation - React components (coming soon)
Nested Agents Integration - Agent orchestration
Audio Fixes Log - Historical audio issues & fixes

Deployment

JETSON_DEPLOYMENT_GUIDE.md - Production server
TV_WEBVIEW_FIX_SUMMARY.md - TV compatibility

Architecture

REFACTORING_SUMMARY.md - Backend structure
FRONTEND_REFACTORING.md - Frontend structure

Recent Changes

Interactive Session Setup (2025-12-04)

Created helper scripts: start-backend.sh, start-frontend.sh, start-webrtc-session.sh
Added comprehensive WebRTC testing guides
Dual logging (console + /tmp/agentic-logs/)

WebRTC Bridge Migration (2025-12-04)

Migrated from Pipecat to pure WebRTC bridge
Direct OpenAI Realtime API connection
Comprehensive unit and integration tests

Documentation Reorganization (2025-12-04)

Extracted detailed content to focused guides
Created QUICK_START.md for immediate setup
Streamlined CLAUDE.md with references

NVM Node Path Enforcement (2025-12-04)

Critical instructions for using full nvm paths
System node may be outdated/incompatible

Database & Memory System (2025-12-02)

MongoDB + ChromaDB fully operational
Memory banks migrated from git history
Comprehensive setup documentation

End of CLAUDE.md

For detailed information on any topic, see the documentation links above.

Last updated: 2025-12-04

FilesExpand file tree

CONTEXT_MEMORY.md

Latest commit

History

CONTEXT_MEMORY.md

File metadata and controls

CLAUDE.md - Agentic System Guide

🚀 Quick Start

Table of Contents

Essential Context

Detect Your Environment First

⚠️ CRITICAL: Node.js Environment Differences

Project Overview

Key Features

Tech Stack

Architecture

Directory Structure

Data Flow

Important URLs

Development Guides

Agent Development

Tool Development

Voice System

Mobile Voice Interface

System Components

Database & Memory

Debugging Tools

Claude Code Integration

Deployment

Jetson Nano Production

Quick Reference

Common Commands

WebRTC Voice Commands

File Locations

Best Practices

Development Workflow

UI Development

Agent Development

Troubleshooting

Backend Issues

Frontend Issues

WebRTC Issues

Documentation Index

Core Documentation

Feature Guides

Voice System

Voice Technical Details

Deployment

Architecture

Recent Changes

Interactive Session Setup (2025-12-04)

WebRTC Bridge Migration (2025-12-04)

Documentation Reorganization (2025-12-04)

NVM Node Path Enforcement (2025-12-04)

Database & Memory System (2025-12-02)