Cerebrum Development Guide

Complete guide for developing, testing, and deploying Cerebrum components.

Development Environment
Project Structure
Sync Workflow
Testing
Debugging
Adding Features
Code Style
Common Tasks

Development Environment

macOS Setup (Primary Development Machine)

Requirements:

macOS (Monterey) or later
VS Code (or VS Code Insiders)
Git
Python 3.11+ (for local testing)
SSH access to CM4 and VPS

Initial Setup:

# Clone repository
cd ~/
git clone https://github.com/artcore-c/Cerebrum.git
cd Cerebrum

# Install local Python env (optional, for linting)
python3 -m venv .venv
source .venv/bin/activate
pip install -r cerebrum-pi/requirements.txt
pip install -r cerebrum-backend/requirements.txt

VS Code Extensions (Recommended):

Python (Microsoft)
Pylance
autopep8 or Black (formatting)
GitLens
Remote - SSH (for direct editing on Pi/VPS)

Project Structure

Cerebrum/                        # Meta-repository (macOS)
│
├── cerebrum-pi/                 # CM4 Orchestrator code
│   ├── cerebrum/                # Python package
│   │   ├── api/                 # FastAPI application
│   │   ├── core/                # VPS client
│   │   └── retrieval/           # Chunking, RAG
│   ├── scripts/                 # Utilities
│   ├── tests/                   # Test suites
│   └── requirements.txt
│
├── cerebrum-backend/            # VPS Backend code
│   ├── vps_server/              # Inference service
│   ├── scripts/                 # Management tools
│   └── requirements.txt
│
├── docs/                        # Documentation
├── scripts/                     # Development tools
│   ├── sync_to_cm4.sh           # Deploy to Raspberry Pi
│   └── sync_to_vps.sh           # Deploy to VPS
│
└── shared/                      # Shared resources
    ├── knowledge_base/
    └── models/

Deployment Targets

Component	Location	Path
CM4 Orchestrator	Raspberry Pi CM4	`/opt/cerebrum-pi/`
VPS Backend	Debian VPS	`~/Cerebrum/cerebrum-backend/`

Sync Workflow

Quick Deploy

Deploy CM4 changes:

./scripts/sync_to_cm4.sh

Deploy VPS changes:

./scripts/sync_to_vps.sh

What Gets Synced

CM4 Sync (sync_to_cm4.sh):

# Syncs:
cerebrum-pi/cerebrum/          # Python code
cerebrum-pi/scripts/           # Utilities
cerebrum-pi/requirements.txt
cerebrum-pi/.env              # If exists locally

# Excludes:
__pycache__/
*.pyc
.venv/
logs/

VPS Sync (sync_to_vps.sh):

# Syncs:
cerebrum-backend/vps_server/
cerebrum-backend/scripts/
cerebrum-backend/requirements.txt
cerebrum-backend/.env         # If exists locally

# Excludes:
__pycache__/
*.pyc
.venv/
logs/
models/                       # Too large, managed separately

Manual Sync (Alternative)

CM4:

rsync -avz --exclude='__pycache__' --exclude='.venv' \
  ~/Cerebrum/cerebrum-pi/ \
  cm4-user@cm4-host:/opt/cerebrum-pi/

VPS:

rsync -avz --exclude='__pycache__' --exclude='.venv' --exclude='models' \
  ~/Cerebrum/cerebrum-backend/ \
  vps-user@vps-host:~/Cerebrum/cerebrum-backend/

Testing

Local Testing (macOS)

Syntax/Import Checks:

# Test CM4 code
cd ~/Cerebrum/cerebrum-pi
python -m py_compile cerebrum/api/main.py
python -m py_compile cerebrum/retrieval/chunker.py

# Test VPS code
cd ~/Cerebrum/cerebrum-backend
python -m py_compile vps_server/main.py

Unit Tests (if available):

cd ~/Cerebrum/cerebrum-pi
pytest tests/

CM4 Testing (On Device)

SSH into CM4:

ssh cm4-user@cm4-host

Full System Test:

cd /opt/cerebrum-pi
./test.sh

Expected output:

Testing CM4 Orchestrator...
✓ Health check passed
✓ VPS connection established
✓ Model listing successful
✓ Inference endpoint responding
All tests passed!

Quick VPS Connection Test:

cd /opt/cerebrum-pi
./test_vps.sh

Manual API Test:

# Health check
curl http://localhost:7000/health | jq

# Simple completion
curl -X POST http://localhost:7000/v1/complete \
  -H "Content-Type: application/json" \
  -d '{"prompt": "def hello():", "language": "python", "max_tokens": 32}' | jq

Watch Logs:

tail -f logs/cerebrum.log

VPS Testing (On Server)

SSH into VPS:

ssh vps-user@vps-host

Full System Test:

cd ~/Cerebrum/cerebrum-backend
./test.sh

Expected output:

Testing VPS Backend...
✓ Health check passed
✓ Model paths verified
✓ Inference endpoint responding
✓ Authentication working
All tests passed!

Manual API Test:

# Health check (no auth)
curl http://127.0.0.1:9000/health | jq

# Model list (requires auth)
curl -H "X-API-Key: $(grep CEREBRUM_API_KEY .env | cut -d= -f2)" \
  http://127.0.0.1:9000/v1/models | jq

Watch Logs:

# If using start.sh
tail -f logs/backend.log

# If using systemd
sudo journalctl -u cerebrum-backend -f

Debugging

Common Issues

CM4: "Can't connect to VPS"

# Check Tailscale
ssh cm4-user@cm4-host
sudo tailscale status

# Test VPS directly
./test_vps.sh

# Check VPS is running
ssh vps-user@vps-host
cd ~/Cerebrum/cerebrum-backend
ps aux | grep uvicorn

CM4: "Port already in use"

ssh cm4-user@cm4-host
sudo lsof -i :7000
# Kill process if needed
kill <PID>

VPS: "Model not found"

ssh vps-user@vps-host

# Verify model paths in code
grep "model_paths" ~/Cerebrum/cerebrum-backend/vps_server/main.py

# Check files exist
ls -lh ~/Cerebrum/cerebrum-backend/models/

VPS: "Out of memory"

ssh vps-user@vps-host

# Check memory
free -h

# Unload models manually
curl -X POST -H "X-API-Key: $(grep CEREBRUM_API_KEY .env | cut -d= -f2)" \
  http://127.0.0.1:9000/v1/cleanup

Debug Logging

Enable verbose logging:

Edit .env on CM4:

LOG_LEVEL=DEBUG

Restart:

./stop.sh
./start.sh

Watch detailed logs:

tail -f logs/cerebrum.log

Request Tracing

Every request gets a UUID for end-to-end tracking:

CM4 logs:

[65652caa-c647-4685-be81-5e51bc97f453] POST /v1/complete/stream 200 182.14s

VPS logs:

[65652caa-c647-4685-be81-5e51bc97f453] Model inference: qwen_7b, 129 tokens

Search logs by request ID:

grep "65652caa-c647-4685-be81-5e51bc97f453" logs/cerebrum.log

Adding Features

Workflow

Branch (optional):

   git checkout -b feature/new-chunking-algo

Edit on macOS:

   code ~/Cerebrum/cerebrum-pi/cerebrum/retrieval/chunker.py

Local syntax check:

   python -m py_compile cerebrum/retrieval/chunker.py

Sync to CM4:

   ./scripts/sync_to_cm4.sh

Test on device:

   ssh cm4-user@cm4-host
   cd /opt/cerebrum-pi
   ./test.sh

Iterate until working
Commit:

   git add cerebrum-pi/cerebrum/retrieval/chunker.py
   git commit -m "feat: improve chunking deduplication algorithm"
   git push origin feature/new-chunking-algo

Example: Adding a New Endpoint

1. Define route (cerebrum-pi/cerebrum/api/routes/inference.py):

@router.post("/v1/explain")
async def explain_code(request: ExplainRequest):
    """Explain what code does"""
    # Implementation
    pass

2. Add schema (cerebrum-pi/cerebrum/api/schemas/):

class ExplainRequest(BaseModel):
    code: str
    language: str

3. Test locally:

python -m py_compile cerebrum/api/routes/inference.py

4. Sync and test:

./scripts/sync_to_cm4.sh
ssh cm4-user@cm4-host
cd /opt/cerebrum-pi
./stop.sh && ./start.sh
curl -X POST http://localhost:7000/v1/explain \
  -H "Content-Type: application/json" \
  -d '{"code": "def fib(n): ...", "language": "python"}'

5. Document (update docs/api/API.md)

Code Style

Python Guidelines

Follow PEP 8:

# Format with autopep8
autopep8 --in-place --aggressive cerebrum/api/main.py

# Or Black (opinionated)
black cerebrum/

Type hints (preferred):

def chunk_text(text: str, max_size: int = 1000) -> List[str]:
    """Chunk text into blocks"""
    pass

Docstrings:

def extract_instruction(prompt: str) -> tuple[str, str]:
    """Extract instruction from prompt.
    
    Args:
        prompt: Full prompt text
        
    Returns:
        Tuple of (code, instruction)
    """
    pass

Project Conventions

File naming:

Modules: snake_case.py
Classes: PascalCase
Functions: snake_case()

Import order:

# Standard library
import os
import sys

# Third-party
from fastapi import FastAPI
import httpx

# Local
from cerebrum.core.vps_client import VPSClient

Constants:

MAX_CONCURRENT_REQUESTS = 2
VPS_ENDPOINT = "http://127.0.0.1:9000"

Common Tasks

Update Dependencies

CM4:

# Edit requirements.txt locally
vim ~/Cerebrum/cerebrum-pi/requirements.txt

# Sync
./scripts/sync_to_cm4.sh

# Install on CM4
ssh cm4-user@cm4-host
cd /opt/cerebrum-pi
source .venv/bin/activate
pip install -r requirements.txt
./stop.sh && ./start.sh

VPS:

# Same process for cerebrum-backend/requirements.txt
./scripts/sync_to_vps.sh
ssh vps-user@vps-host
cd ~/Cerebrum/cerebrum-backend
source .venv/bin/activate
pip install -r requirements.txt
sudo systemctl restart cerebrum-backend

Add a New Model

1. Download model to VPS:

ssh vps-user@vps-host
cd ~/Cerebrum/cerebrum-backend/models/
wget https://huggingface.co/.../model.gguf

2. Update model map (vps_server/main.py):

model_paths = {
    "qwen_7b": "/home/unicorn1/Cerebrum/cerebrum-backend/models/qwen-7b-q4.gguf",
    "new_model": "/home/unicorn1/Cerebrum/cerebrum-backend/models/model.gguf",  # ADD THIS
}

3. Restart VPS:

sudo systemctl restart cerebrum-backend

4. Update CM4 routing (cerebrum-pi/cerebrum/api/routes/_chunking_helper.py):

_MODEL_MAP = {
    "python": "qwen_7b",
    "haskell": "new_model",  # ADD THIS
}

5. Sync and restart CM4:

./scripts/sync_to_cm4.sh
ssh cm4-user@cm4-host
cd /opt/cerebrum-pi
./stop.sh && ./start.sh

Regenerate API Key

On VPS:

ssh vps-user@vps-host
cd ~/Cerebrum/cerebrum-backend/scripts
./generate_api_key.sh
# Copy new key

Update CM4 .env:

ssh cm4-user@cm4-host
nano /opt/cerebrum-pi/.env
# Paste new CEREBRUM_API_KEY

Restart both:

# VPS
sudo systemctl restart cerebrum-backend

# CM4
cd /opt/cerebrum-pi
./stop.sh && ./start.sh

Performance Profiling

CM4 Overhead Measurement

ssh cm4-user@cm4-host
cd /opt/cerebrum-pi

# Time a simple request
time curl -X POST http://localhost:7000/v1/complete \
  -H "Content-Type: application/json" \
  -d '{"prompt": "def test():", "language": "python", "max_tokens": 32}'

Check logs for breakdown:

Chunking large prompt: 50ms
VPS request: 15.2s
Total: 15.25s

VPS Resource Monitoring

ssh vps-user@vps-host

# Watch resources during inference
watch 'curl -s http://127.0.0.1:9000/health | jq'

# System stats
htop

Git Workflow

Branching Strategy

main            - Production-ready code
├── dev         - Integration branch
└── feature/*   - New features
└── fix/*       - Bug fixes

Commit Messages

feat: add semantic search with FAISS
fix: resolve circuit breaker timeout issue
docs: update API reference for /v1/explain
refactor: extract chunking helper logic
test: add unit tests for instruction parser

Before Committing

# Run tests
cd ~/Cerebrum/cerebrum-pi
pytest tests/

# Check syntax
python -m py_compile cerebrum/**/*.py

# Format code
black cerebrum/

# Commit
git add .
git commit -m "feat: improve chunking algorithm"
git push

Troubleshooting Development Issues

"rsync: command not found"

# Install on macOS
brew install rsync

"Permission denied" on sync

# Fix CM4 permissions
ssh cm4-user@cm4-host
sudo chown -R $(id -un):$(id -gn) /opt/cerebrum-pi

Python import errors after sync

# Reinstall dependencies
ssh cm4-user@cm4-host
cd /opt/cerebrum-pi
source .venv/bin/activate
pip install --force-reinstall -r requirements.txt

Can't SSH to CM4/VPS

# Check Tailscale
tailscale status

# Use IP instead
ssh cm4-user@cm4-host
ssh vps-user@vps-host

Best Practices

Always test locally first (syntax checks)
Sync incrementally (one component at a time)
Watch logs during testing (tail -f logs/)
Use request IDs for debugging (grep by UUID)
Keep .env files in sync (CM4 ↔ VPS API keys must match)
Don't sync models (too large, manage separately)
Restart services after config changes
Document breaking changes (update API.md)
Test on actual hardware (Mac ≠ ARM CM4)
Commit working code only (test before push)

Resources

Main README: /README.md
API Reference: /docs/api/API.md
Architecture: /docs/architecture/ARCHITECTURE.md
CM4 Setup: /cerebrum-pi/README.md
VPS Setup: /cerebrum-backend/README.md

FilesExpand file tree

DEVELOPMENT.md

Latest commit

History