A self-evolving AI assistant that learns, adapts, and grows with your household.
Hardware-agnostic Β· Privacy-first Β· Family-safe Β· Self-learning
Atlas Cortex transforms a local LLM into an intelligent home assistant that understands who's speaking, adapts to each family member, controls your smart home, and gets smarter every day β all running on your hardware, with zero cloud dependencies.
| Layer | Latency | What Happens |
|---|---|---|
| Context Assembly | ~1ms | Identifies speaker, room, sentiment, time-of-day |
| Instant Answers | ~5ms | Date, time, math, greetings β no LLM needed |
| Plugin Dispatch | ~100ms | Smart home control, lists, knowledge search |
| LLM Generation | ~500β4000ms | Full reasoning with filler streaming for zero perceived wait |
- Natural language control β "Turn off the bedroom lights" executes directly via Home Assistant API
- Spatial awareness β knows which room you're in via satellite mics and presence sensors
- Scene automation β "Good night" triggers your bedtime routine
- Self-learning β commands that go to the LLM are analyzed nightly and converted into fast regex patterns
- Multi-TTS stack β Qwen3-TTS (primary), Fish Audio S2 (story character voices), Orpheus (emotional), Kokoro (CPU), Piper (fast fallback)
- Voice identification β recognizes family members by voice, personalizes responses per person
- TTS hot-swap β swap voice models at runtime for character voices in stories
- Sentence-boundary streaming β starts speaking before the full response is generated
- Speaker age estimation β hybrid age detection for content tier assignment
- Animated face β Nick Jr. Face-inspired SVG avatar with real-time lip-sync and expressions
- Audio-synced visemes β mouth shapes timed to actual TTS audio duration, not guesswork
- Content-aware expressions β silly grin after joke punchlines, thinking face during processing
- Pre-cached joke bank β 25 kid-friendly jokes with instant TTS playback, weekly rotation
- Interactive button β
#show-joke-buttonURL hash enables a "Tell me a joke" button on the display - WebSocket streaming β TTS audio, visemes, and expressions all delivered in real-time
Neutral Β· Talking Β· Silly grin (after a joke punchline)
- Natural language time parsing β "Wake me up at 7am", "Set a 15-minute timer"
- Recurring alarms with RRULE recurrence rules
- Notification routing β deliver to the right satellite speaker or HA media player
- Snooze/dismiss via voice β "Snooze for 5 minutes" / "Stop"
- Built-in routines β "Good morning" β lights, coffee, weather, calendar
- Custom routines β "When I say 'movie time', dim lights and turn on TV"
- Multiple triggers β voice command, cron schedule, HA events
- Conversational builder β Atlas asks clarifying questions to build routines
- Templates β pre-built routines for common scenarios
- Rule engine with notification throttling
- Weather, energy, anomaly, and calendar providers
- Daily briefing β personalized morning summary
- Pattern detection β learns household habits over time
- Socratic tutoring β guides understanding, never gives direct answers
- Quiz generator β math curriculum through Calculus III
- 3 STEM games β Number Quest, Science Safari, Word Wizard
- Progress tracking and parent reports per child
- Targeted announce β "Tell the kids dinner is ready" β children's rooms
- Whole-house broadcast β all satellites simultaneously
- Two-way calling β "Atlas, talk to the garage"
- Drop-in monitoring β listen to a room without announcement
- Zone groups β broadcast to custom room groups
- YouTube Music, Plex, Audiobookshelf, Podcasts
- Local music library with ID3 tag scanning
- Multi-room playback via satellite β Chromecast β HA media players
- Playback router β picks the best output for each room
- Interactive branching stories with user choices
- Character voices via Fish Audio S2 with TTS hot-swap
- Story library β save, resume, and replay stories
- Age-appropriate content per user profile
- Conversation quality analysis β identifies areas for improvement
- LoRA training pipeline on AMD GPUs (ROCm) β runs overnight
- Model scout with safety gates β finds and evaluates new models
- A/B testing for model comparisons
- Personality drift monitoring β ensures consistency over time
- Interactive chat:
atlas chat - Autonomous agent:
atlas agent "build an API" - 31 agent tools β file, shell, git, web, vision, memory
- ReAct reasoning loop with context management and sessions
- Browser chat with WebSocket streaming
- Voice input/output in the browser
- Animated avatar with lip-sync
- Unified dashboard for all Atlas features
- Age-appropriate responses β automatically adapts vocabulary and content for toddlers, children, teens, and adults
- Educational mode β uses scientific terminology for biology/anatomy at all ages β never evasive
- 5-layer jailbreak defense β regex patterns, semantic analysis, system prompt hardening, output monitoring, adaptive learning
- PII protection β SSN, credit card, phone, email auto-redacted from logs and memory
- Crisis detection β recognizes self-harm/emergency language and responds with appropriate resources
- HOT/COLD architecture β ChromaDB vector search + SQLite FTS5 with reciprocal rank fusion
- Persistent memory β remembers conversations, preferences, and facts across sessions
- Nightly evolution β analyzes patterns, learns from mistakes, evolves personality profiles
- Anti-hallucination β confidence scoring, grounding loops, and mistake tracking
- Per-user adaptation β vocabulary level, preferred tone, communication style
- Honest personality β pushes back on bad ideas, challenges in tutoring mode, never sycophantic
- Emotional evolution β builds unique rapport with each household member over time
- Parental controls β content filtering, allowed hours, restricted actions per child
- Dark-themed dashboard β real-time stats, recent activity, system health at a glance
- 20 views β Chat, Dashboard, Users, UserDetail, Parental, Safety, Voice, Avatar, Devices, Satellites, SatelliteDetail, Plugins, Scheduling, Routines, Learning, Proactive, Media, Intercom, Evolution, Stories, System
- User management β profiles, age settings, vocabulary levels, parental controls
- Safety monitoring β guardrail event log, jailbreak pattern management, content tier overrides
- Plugin management β enable/disable 21 built-in plugins with per-plugin settings
- Voice enrollment β view and manage speaker profiles, confidence thresholds
- Device management β Home Assistant devices, aliases, command patterns
- Evolution tracking β rapport scores, emotional profiles, nightly evolution logs
- System overview β hardware info, GPU assignment, model configs, discovered services
π Full Admin Panel Guide β β walkthrough with screenshots of every page
βββββββββββββββββββββββββ
β User Interface β
β Open WebUI / Voice β
β / Satellite Mics β
βββββββββββββ¬ββββββββββββ
β
βββββββββββββΌββββββββββββ
β Atlas Cortex β
β Server (:5100) β
β OpenAI-compatible β
βββββββββββββ¬ββββββββββββ
β
βββββββββββββββββββββββββΌββββββββββββββββββββββββ
β β β
βββββββββββΌβββββββββββ βββββββββββΌβββββββββββ βββββββββββΌβββββββββββ
β Input Pipeline β β Safety Guardrails β β Voice Engine β
β β β β β β
β L0: Context (1ms) β β β’ Content tiers β β β’ Qwen3-TTS β
β L1: Instant (5ms) β β β’ Jailbreak defense β β β’ Fish Audio S2 β
β L2: Plugins (100ms) β β β’ PII redaction β β β’ Orpheus / Kokoro β
β L3: LLM (500ms+) β β β’ Crisis detection β β β’ Piper fallback β
βββββββββββ¬βββββββββββ βββββββββββββββββββββββ βββββββββββββββββββββββ
β
βββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Integrations β
β β
β π Home Assistant π Lists π Knowledge π Memory β
β π§ Discovery π¦ Backup π Learning β° Scheduling β
β π Routines π‘ Proactive π’ Intercom π΅ Media β
β π Stories π€ Evolution π» CLI Agent π Web App β
βββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββ
β
ββββββββββββΌββββββββββββ
β SQLite + ChromaDB β
β (WAL mode, 50+ tbl) β
ββββββββββββ¬ββββββββββββ
β
ββββββββββββΌββββββββββββ
β Nightly Evolution β
β LoRA training β
β Model scouting β
β Pattern learning β
β Profile evolution β
β Device discovery β
ββββββββββββββββββββββββ
Atlas detects all GPUs at startup and assigns optimal roles:
βββββββββββββββββββ βββββββββββββββββββ βββββββββββββββββββ
β GPU 0 (Largest)β β GPU 1 (Second) β β iGPU (Fallback)β
β ββββββββββββββββ β ββββββββββββββββ β ββββββββββββββββ
β LLM Inference β β Voice / TTS β β Lightweight β
β Transformers β β Qwen3-TTS β β tasks only β
β 20GB+ VRAM β β 8-12GB VRAM β β β
βββββββββββββββββββ βββββββββββββββββββ βββββββββββββββββββ
- Supported GPUs: AMD (ROCm), NVIDIA (CUDA), Intel (oneAPI/IPEX), Apple (Metal)
- Auto-sizing: Models selected based on available VRAM β from 1.7B (4GB) to 72B (48GB+)
- Mixed vendors: Run AMD + Intel GPUs in the same system via separate containers
- Python 3.11+
- HuggingFace Transformers β models download automatically on first use
- Open WebUI v0.8.5+ (recommended) or any OpenAI-compatible client
Note: Ollama is still supported as a legacy fallback β set
LLM_PROVIDER=ollamato use it.
# Clone the repository
git clone https://github.com/Betanu701/atlas-cortex.git
cd atlas-cortex
# Start with Docker Compose (includes Qwen3-TTS, Whisper, Kokoro, Piper, and more)
# HuggingFace models download automatically via cached volume
docker compose -f docker/docker-compose.yml up -d
# For NVIDIA GPU support:
docker compose -f docker/docker-compose.yml -f docker/docker-compose.gpu-nvidia.yml up -d
# Atlas is now running at http://localhost:5100
# Chat UI: http://localhost:5100/chat
# Admin panel: http://localhost:5100/admin/# Clone and set up
git clone https://github.com/Betanu701/atlas-cortex.git
cd atlas-cortex
# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate
# Install dependencies (includes transformers, accelerate, sentence-transformers)
pip install -r requirements.txt
# Run the interactive installer
python -m cortex.install
# Or start the server directly (HuggingFace models download on first use)
CAG_MODEL=Qwen/Qwen3-4B EMBED_MODEL=all-MiniLM-L6-v2 python -m cortex.server- Open your Open WebUI instance
- Go to Admin β Settings β Connections
- Add a new OpenAI-compatible connection:
- URL:
http://<atlas-host>:5100/v1 - Model:
atlas-cortex
- URL:
- Start chatting β Atlas handles the rest
The admin panel is a Vue 3 SPA served directly from the Atlas Cortex server.
# Build the admin panel (requires Node.js 18+)
cd admin
npm install
npx vite build
cd ..
# Start the server (admin panel is now available)
python -m cortex.serverOpen http://localhost:5100/admin/ in your browser.
| Default username | admin |
| Default password | atlas-admin |
β οΈ Change the default password immediately via the admin panel.π Full Admin Panel Guide β β walkthrough of every page with screenshots
π API Reference (144+ endpoints) β β complete endpoint documentation
# Scan your network for Home Assistant, Nextcloud, MQTT, etc.
python -m cortex.discoverAtlas finds available services on your network and configures integrations automatically.
| Variable | Default | Description |
|---|---|---|
CORTEX_HOST |
0.0.0.0 |
Server bind address |
CORTEX_PORT |
5100 |
Server port |
CORTEX_DATA_DIR |
./data |
Database and state directory |
LLM_PROVIDER |
transformers |
LLM backend (transformers, ollama, openai_compatible) |
CAG_MODEL |
Qwen/Qwen3-4B |
HuggingFace model for inference (Transformers provider) |
CAG_DEVICE |
auto |
Device for model inference (auto, cuda, cpu) |
CAG_DTYPE |
auto |
Model dtype (auto, float16, bfloat16) |
EMBED_MODEL |
all-MiniLM-L6-v2 |
Sentence-transformers embedding model |
OLLAMA_BASE_URL |
http://localhost:11434 |
Ollama API URL (legacy fallback) |
OPENAI_BASE_URL |
β | Custom OpenAI-compatible endpoint |
OPENAI_API_KEY |
β | API key for OpenAI-compatible backends |
MODEL_FAST |
qwen2.5:14b |
Model for quick factual answers (Ollama/OpenAI providers) |
MODEL_THINKING |
qwen3:30b-a3b |
Model for complex reasoning (Ollama/OpenAI providers) |
HA_URL |
β | Home Assistant URL (e.g., http://192.168.1.100:8123) |
HA_TOKEN |
β | Home Assistant long-lived access token |
CORTEX_JWT_SECRET |
atlas-cortex-change-me |
Secret key for admin JWT tokens (change in production!) |
CORTEX_JWT_EXPIRY |
86400 |
Admin session duration in seconds (default: 24 hours) |
Atlas exposes an OpenAI-compatible API so any client that works with OpenAI/Ollama works with Atlas.
# Streaming
curl http://localhost:5100/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "atlas-cortex",
"messages": [{"role": "user", "content": "Turn off the living room lights"}],
"stream": true
}'# Generate speech with emotion
curl http://localhost:5100/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "kokoro",
"input": "Good morning! The weather looks beautiful today.",
"voice": "af_bella"
}' --output speech.wav
# List available voices
curl http://localhost:5100/v1/audio/voicescurl http://localhost:5100/healthAll admin endpoints require a JWT token. Quick example:
# Login and get token
TOKEN=$(curl -s http://localhost:5100/admin/auth/login \
-H "Content-Type: application/json" \
-d '{"username": "admin", "password": "atlas-admin"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
# Use with any admin endpoint
curl http://localhost:5100/admin/dashboard \
-H "Authorization: Bearer $TOKEN"atlas-cortex/
βββ admin/ # Vue 3 Admin Panel (SPA, 20 views)
β βββ src/
β β βββ views/ # 20 admin views (dashboard, users, safety, etc.)
β β βββ components/ # Reusable components (NavBar, DataTable, etc.)
β β βββ router/ # Vue Router with auth guards
β β βββ stores/ # Pinia auth store
β β βββ api.js # API client with JWT handling
β βββ package.json
β βββ vite.config.js
βββ cortex/ # Core Python package
β βββ server.py # OpenAI-compatible FastAPI server
β βββ auth.py # JWT authentication (bcrypt + PyJWT)
β βββ pipe.py # Open WebUI Pipe function
β βββ db.py # SQLite schema (50+ tables, WAL mode)
β βββ admin/ # Admin API domain routers (9 sub-routers)
β βββ pipeline/ # 4-layer processing pipeline
β β βββ layer0_context.py # Context assembly, sentiment, spatial
β β βββ layer1_instant.py # Instant answers (math, date, identity)
β β βββ layer2_plugins.py # Plugin dispatch (21 built-in plugins)
β β βββ layer3_llm.py # Filler streaming + LLM generation
β βββ providers/ # LLM backend abstraction
β β βββ transformers.py # HuggingFace Transformers provider (default)
β β βββ ollama.py # Ollama provider (legacy fallback)
β β βββ openai_compat.py # Any OpenAI-compatible backend
β βββ speech/ # All audio synthesis/transcription
β β βββ tts.py # Multi-provider TTS with hot-swap
β β βββ stt.py # Whisper + Wyoming STT
β β βββ voices.py # Voice resolution (satelliteβuserβsystem)
β β βββ cache.py # Unified audio cache
β β βββ fish_audio.py # Fish Audio S2 character voices
β β βββ hotswap.py # Runtime TTS model swapping
β βββ orchestrator/ # Request coordination (STTβpipelineβTTS)
β βββ voice/ # Legacy voice module (β cortex.speech)
β βββ safety/ # Safety guardrails + jailbreak defense
β βββ plugins/ # Plugin framework (CortexPlugin, registry)
β βββ integrations/ # HA, knowledge, lists, learning
β βββ memory/ # HOT/COLD memory (BM25 + vector, RRF)
β βββ profiles/ # User profiles & parental controls
β βββ context/ # Context window management
β βββ avatar/ # SVG avatar, lip-sync, expressions
β βββ filler/ # Sentiment-aware filler streaming
β βββ grounding/ # Anti-hallucination engine
β βββ backup/ # Automated backup/restore + offsite
β βββ scheduling/ # Alarms, timers, reminders (NL time parsing)
β βββ routines/ # Routine automations, triggers, templates
β βββ proactive/ # Proactive intelligence, daily briefing
β βββ learning/ # Self-learning (fallthrough analysis)
β βββ intercom/ # Announce, broadcast, two-way calling
β βββ media/ # YouTube Music, Plex, ABS, podcasts
β βββ stories/ # Story generator, character voices
β βββ evolution/ # LoRA training, model scout, drift
β βββ cli/ # Atlas CLI agent (REPL, 31 tools)
β βββ notifications/ # Alert and notification routing
β βββ selfmod/ # Self-evolution security gates
β βββ content/ # Pre-cached content (jokes)
β βββ scheduler/ # Background task management
β βββ satellite/ # Satellite WebSocket + provisioning
β βββ install/ # Hardware detection & installer
β βββ discovery/ # Network service discovery
β βββ integrity/ # Data integrity checks
βββ satellite/ # Satellite agent (Pi, ESP32)
βββ docker/ # Docker deployment configs
βββ docs/ # Architecture, roadmap, guides (35+ files)
βββ mocks/ # GPU-free development servers
βββ seeds/ # Initial data (command patterns)
βββ tests/ # 3,660+ tests
βββ requirements.txt
βββ pytest.ini
# Run all tests
source .venv/bin/activate
python -m pytest tests/ -q
# Run specific module
python -m pytest tests/test_pipeline.py -v
python -m pytest tests/test_safety.py -v
python -m pytest tests/test_voice.py -vCurrent status: 3,660+ tests passing across pipeline, providers, safety, voice, discovery, integrations, filler, memory, learning, evolution, avatar, admin, spatial, Wyoming, WebDAV, CalDAV, backup, multi-room, scheduling, routines, proactive, intercom, media, stories, CLI, and web app modules.
| Phase | Module | Status | Description |
|---|---|---|---|
| C0 | Installer & Backend | β Complete | LLM provider abstraction, GPU detection, CLI installer |
| C1 | Core Pipeline | β Complete | 4-layer pipeline, sentiment, instant answers, filler streaming |
| C3a | Voice Identity | β Complete | Speaker recognition, enrollment, hybrid age estimation |
| C4 | Emotional Evolution | β Complete | Rapport tracking, personality drift, proactive suggestions |
| C5 | Memory System | β Complete | HOT/COLD paths, vector search, BM25, RRF fusion |
| C6 | User Profiles | β Complete | Age-awareness, onboarding, parental controls |
| C7 | Avatar System | β Complete | SVG face, lip-sync visemes, emotion expressions, joke bank, TTS caching |
| C9 | Backup & Restore | β Complete | Automated nightly backups, one-command restore |
| C10 | Context Management | β Complete | Context windows, compaction, overflow recovery |
| C11 | Voice & Speech | β Complete | Multi-TTS (Qwen3-TTS, Fish Audio S2, Orpheus, Kokoro, Piper), hot-swap, streaming |
| C12 | Safety Guardrails | β Complete | Content tiers, jailbreak defense, PII redaction |
| β | Admin Web Panel | β Complete | Vue 3 dashboard (20 views), JWT auth, 30+ REST endpoints |
| Phase | Module | Status | Description |
|---|---|---|---|
| I1 | Service Discovery | β Complete | HTTP-probe scanner, service registry, config wizard |
| I2 | Home Assistant | β Complete | REST + WebSocket client, device bootstrap, patterns |
| I3 | Voice Pipeline | β Complete | Wyoming STT/TTS, spatial awareness, multi-room commands |
| I4 | Self-Learning | β Complete | Fallthrough analysis, pattern lifecycle, nightly evolution |
| I5 | Knowledge Sources | β Complete | WebDAV/Nextcloud, CalDAV calendars, sync scheduler |
| I6 | List Management | β Complete | Multi-backend lists, HA to-do discovery, permissions |
| I7 | Offsite Backup | β Complete | NAS rsync/SMB, voice commands, retention policy |
| Phase | Module | Status | Description |
|---|---|---|---|
| S2.5 | Satellite System | βΈοΈ Wake word deferred | Distributed speakers/mics, Pi + ESP32, provisioning |
| P2.7 | Fast-Path Plugins | β Complete | 21 built-in plugins (weather, dictionary, wikipedia, conversions, movie, cooking, news, translation, stocks, sports, sound library, scheduling, routines, daily briefing, STEM games, stories, intercom, media + core 3) |
| Phase | Module | Status | Description |
|---|---|---|---|
| P3 | Alarms, Timers & Reminders | β Complete | NL time parser, RRULE recurrence, notification routing |
| P4 | Routines & Automations | β Complete | Voice/cron/HA triggers, templates, conversational builder |
| P5 | Proactive Intelligence | β Complete | Rule engine, throttle, weather/energy/anomaly/calendar providers, daily briefing |
| P6 | Learning & Education | β Complete | Socratic tutoring, quiz gen (through Calc III), 3 STEM games, progress tracking |
| P7 | Intercom & Broadcasting | β Complete | Announce, broadcast, zones, two-way calling, drop-in monitoring |
| P8 | Media & Entertainment | β Complete | YouTube Music, Plex, Audiobookshelf, podcasts, local library, playback router |
| P9 | Self-Evolution | β Complete | Conversation analysis, LoRA training (ROCm/AMD), model scout, A/B testing, drift monitor |
| P10 | Story Time | β Complete | Generator, character voices (Fish Audio S2), TTS hot-swap, interactive stories, library |
| P11 | Atlas CLI Agent | β Complete | REPL, 31 agent tools, ReAct loop, context management, sessions |
| P12 | Standalone Web App | β Complete | Browser chat, WebSocket streaming, voice I/O, avatar, unified dashboard |
| Part | Name | Description |
|---|---|---|
| 13 | Legacy Protocol | Backward compatibility for older integrations |
| 14 | Household Management | Pet care, cooking assistant, inventory, chore tracking |
| 15 | Security & Monitoring | Camera summaries, motion alerts, visitor history |
| 16 | Health & Wellness | Medication reminders, sleep analysis, air quality |
| 17 | Multi-Language Support | Real-time language detection, per-user preferences |
| 18 | Visual Media & Casting | Screen casting, photo display, visual content |
π Full roadmap with detailed plans: docs/roadmap.md
Comprehensive design documentation lives in the docs/ directory:
| Document | Description |
|---|---|
| Quick Start | 5-minute setup guide (Docker, bare metal, satellite) |
| Configuration | All environment variables and settings |
| Architecture | System design, pipeline layers, evolution engine |
| Data Model | 50+ SQLite tables, normalized schema, relationships |
| Voice Engine | TTS providers, emotion composition, streaming |
| Safety Guardrails | Content tiers, jailbreak defense, crisis protocol |
| Context Management | Context windows, compaction, multi-GPU detection |
| Memory System | HOT/COLD architecture, vector search, RRF fusion |
| User Profiles | Age-awareness, onboarding, parental controls |
| Personality | Honesty system, pushback, tutoring mode |
| Grounding | Anti-hallucination, confidence scoring |
| Knowledge Access | Document indexing, privacy gates |
| Lists | Multi-backend lists, permissions |
| Avatar System | Lip-sync, visemes, emotion expressions |
| Backup & Restore | Automated backups, one-command restore |
| Admin Panel Guide | Full walkthrough with screenshots |
| API Reference | All 144+ API endpoints |
| Plugin Guide | Plugin system for admins and developers |
| Persistent Storage | Docker volumes and data paths |
| Satellite System | Satellite speaker/mic architecture |
| Alarms & Timers | NL time parsing, recurrence, notification routing |
| Routines | Triggers, templates, conversational builder |
| Proactive Intelligence | Rule engine, providers, daily briefing |
| Learning & Education | Tutoring, quizzes, STEM games |
| Intercom | Announce, broadcast, two-way calling |
| Media | YouTube Music, Plex, Audiobookshelf, podcasts |
| Roadmap | Future features and implementation plan |
| Phases | Implementation roadmap and dependency graph |
| Installation | Installer flow, backend abstraction |
Atlas Cortex is open source and welcomes contributions!
- Fork the repository
- Create a feature branch:
git checkout -b feature/my-feature - Make your changes and add tests
- Run the test suite:
python -m pytest tests/ -q - Submit a Pull Request
git clone https://github.com/Betanu701/atlas-cortex.git
cd atlas-cortex
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m pytest tests/ -q # verify everything worksMIT β use it, modify it, build on it.

