Skip to content

Latest commit

 

History

History

ruvLLM

README.md

RuvLLM

Rust License Tests CPU HuggingFace

Self-Optimizing Neural Architecture (SONA) with LFM2 Cortex, Ruvector Memory, and Intelligent Routing

"The intelligence is not in one model anymore. It is in the loop."

What is RuvLLM?

RuvLLM is a self-learning language model orchestration system that combines frozen foundation models with adaptive memory and intelligent routing. Unlike traditional LLMs that rely solely on static parameters, RuvLLM continuously improves from every interaction through three temporal learning loops.

Key Innovation: RuvLLM doesn't replace your LLM—it makes any LLM smarter over time by learning from experience, routing intelligently, and preventing catastrophic forgetting.

┌─────────────────────────────────────────────────────────────────────────┐
│                         RuvLLM Architecture                              │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│    Query ──► Embedding ──► Memory Search ──► Router Decision            │
│                               │                    │                     │
│                               ▼                    ▼                     │
│                         Graph Attention      Model Selection             │
│                               │                    │                     │
│                               └────────┬───────────┘                     │
│                                        ▼                                 │
│                              ┌─────────────────────┐                     │
│                              │   LLM Inference    │                     │
│                              │  (Any LLM Backend)  │                     │
│                              └─────────────────────┘                     │
│                                        │                                 │
│                                        ▼                                 │
│                    ┌───────────────────────────────────┐                │
│                    │  SONA Learning (3 Temporal Loops) │                │
│                    │  • Instant: Per-request MicroLoRA │                │
│                    │  • Background: Hourly patterns    │                │
│                    │  • Deep: Weekly EWC++ updates     │                │
│                    └───────────────────────────────────┘                │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Features

Core Components

Component Description Implementation
LFM2 Cortex Frozen reasoning engine (135M-2.6B params) Mock, Candle, or external (llama.cpp/vLLM)
Ruvector Memory Adaptive synaptic mesh with HNSW indexing Full CPU implementation with graph expansion
FastGRNN Router Intelligent model selection circuit Sparse + low-rank matrices with EWC learning
Graph Attention Multi-head attention with edge features 8-head attention, layer normalization
SONA Engine Self-optimizing neural architecture LoRA + EWC++ + ReasoningBank

SONA: Self-Optimizing Neural Architecture

RuvLLM introduces SONA, a three-tier temporal learning system:

┌──────────────────────────────────────────────────────────────────────────┐
│  Loop A: Instant (Per-Request)                           Latency: <100μs │
│  ──────────────────────────────────────                                  │
│  • Records query trajectories with activation patterns                   │
│  • MicroLoRA adaptation (rank 1-2) for immediate improvement             │
│  • SIMD-optimized: 2,236 ops/sec throughput                              │
├──────────────────────────────────────────────────────────────────────────┤
│  Loop B: Background (Hourly)                                             │
│  ─────────────────────────────                                           │
│  • K-means++ clustering extracts patterns (100 clusters = 1.3ms search)  │
│  • Base LoRA updates (rank 4-16) from successful patterns                │
│  • ReasoningBank stores learned strategies                               │
├──────────────────────────────────────────────────────────────────────────┤
│  Loop C: Deep (Weekly)                                                   │
│  ─────────────────────                                                   │
│  • Dream consolidation across all memory                                 │
│  • EWC++ prevents catastrophic forgetting (λ=2000 optimal)               │
│  • Concept hierarchies created, old nodes archived                       │
└──────────────────────────────────────────────────────────────────────────┘

Advanced Features

Feature Description
SIMD Inference Native AVX2/AVX512/SSE4.1 operations for CPU optimization
Q4 Quantization 4-bit weight quantization for memory efficiency
MicroLoRA Per-request adaptation with rank 1-2 (benchmark: rank-2 is 5% faster)
EWC++ Enhanced elastic weight consolidation with online Fisher estimation
ReasoningBank Pattern storage with K-means++ clustering
HuggingFace Export Export LoRA weights, patterns, and preference pairs
Real Inference Candle-based inference with HuggingFace model support
Multi-Model Routing Automatic selection between SmolLM, Qwen2, TinyLlama
Federated Learning Distributed learning across ephemeral agents with central coordinator
WASM Support Run SONA in browsers and edge devices
Training Pipelines Templated training for code, chat, reasoning, and custom agents
Agent Factory Create and manage multiple specialized learning agents

Federated Learning Architecture

RuvLLM supports federated learning where ephemeral agents collect trajectories and export to a central coordinator:

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Agent A    │     │  Agent B    │     │  Agent C    │
│ (ephemeral) │     │ (ephemeral) │     │ (ephemeral) │
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       │                   │                   │
       │    export()       │    export()       │    export()
       ▼                   ▼                   ▼
  ┌────────────────────────────────────────────────┐
  │            Federated Coordinator               │
  │         (persistent, large capacity)           │
  │  • Aggregates trajectories from all agents     │
  │  • Quality-filtered acceptance (threshold)     │
  │  • Auto-consolidation every N agents           │
  │  • Shares patterns with new agents             │
  └────────────────────────────────────────────────┘

Key Components:

  • EphemeralAgent: Short-lived agents that process tasks and export learned state
  • FederatedCoordinator: Central aggregator with 50K trajectory capacity
  • AgentExport: Serializable state containing trajectories, stats, and patterns
  • Quality Filtering: Only high-quality trajectories (>0.4 score) are aggregated

Performance Benchmarks

Orchestration Latency (CPU-Only)

Metric Value Notes
Initialization 3.71ms Full system startup
Average Query 0.09ms Single query latency
Session Query 0.04ms With context reuse
Throughput ~38,000 q/s 8 concurrent queries
Memory Footprint ~50MB Base system

Latency Breakdown

Embedding:    ~0.02ms  ████░░░░░░  (20%)
Retrieval:    ~0.01ms  ██░░░░░░░░  (10%)
Routing:      ~0.01ms  ██░░░░░░░░  (10%)
Attention:    ~0.02ms  ████░░░░░░  (20%)
Generation:   ~0.04ms  ████████░░  (40%)

SONA Learning Performance

Component Metric Value
MicroLoRA Throughput 2,236 ops/sec
MicroLoRA Batch-32 Latency 0.447ms
ReasoningBank Pattern Search 1.3ms (100 clusters)
EWC++ Fisher Update <1ms

Comparison with Traditional Systems

System P50 (ms) P95 (ms) vs GPT-4o
GPT-4o (API) 450.00 585.00 1.0x (baseline)
Claude 3.5 Sonnet 380.00 456.00 1.2x
Gemini 2.0 Flash 180.00 234.00 2.5x
Llama 3.3 70B (vLLM) 120.00 168.00 3.8x
RuvLLM Orchestration 0.06 0.08 ~7,500x

Note: RuvLLM orchestration latency measures memory retrieval, routing, and context preparation—NOT LLM generation. Actual response quality depends on your LLM backend.

Feature Comparison

Feature GPT-4o Claude RAG vLLM RuvLLM
On-device Inference ✗ ✗ ✗ ✓ ✓
Continuous Learning ✗ ✗ ✗ ✗ ✓
Graph-based Memory ✗ ✗ △ ✗ ✓
Adaptive Model Routing ✗ ✗ ✗ ✗ ✓
EWC Anti-Forgetting ✗ ✗ ✗ ✗ ✓
LoRA Adaptation ✗ ✗ ✗ ✗ ✓
Pattern Extraction ✗ ✗ ✗ ✗ ✓
HuggingFace Export ✗ ✗ ✗ ✗ ✓
SIMD Optimization ✗ ✗ ✗ △ ✓
Sub-ms Orchestration ✗ ✗ ✗ ✗ ✓
Federated Learning ✗ ✗ ✗ ✗ ✓
WASM/Browser Support ✗ ✗ ✗ ✗ ✓
Training Pipelines ✗ ✗ ✗ ✗ ✓
Works with ANY LLM ✗ ✗ ✓ ✗ ✓

Legend: ✓ = Full Support, △ = Partial, ✗ = Not Supported

Quick Start

Prerequisites

  • Rust 1.77+
  • Cargo

Installation

# Clone the repository
git clone https://github.com/ruvnet/ruvector.git
cd ruvector/examples/ruvLLM

# Build in release mode
cargo build --release

Run the Demo

# Interactive demo with mock inference
cargo run --bin ruvllm-demo --release

# SIMD capabilities demo
cargo run --bin ruvllm-simd-demo --release

# Quick benchmark
cargo run --bin ruvllm-bench --release

# Full benchmark suite
cargo run --bin ruvllm-benchmark-suite --release

# HTTP server (requires 'server' feature)
cargo run --bin ruvllm-server --release --features server

# Pretraining pipeline
cargo run --bin ruvllm-pretrain --release

# HuggingFace export (requires 'hf-export' feature)
cargo run --bin ruvllm-export --release --features hf-export -- help

Library Usage

use ruvllm::{Config, RuvLLM, Result};

#[tokio::main]
async fn main() -> Result<()> {
    // Configure the system
    let config = Config::builder()
        .embedding_dim(768)
        .router_hidden_dim(128)
        .hnsw_params(32, 200, 64)  // M, ef_construction, ef_search
        .learning_enabled(true)
        .build()?;

    // Initialize
    let llm = RuvLLM::new(config).await?;

    // Create a session for multi-turn conversation
    let session = llm.new_session();

    // Query with session context
    let response = llm.query_session(&session, "What is machine learning?").await?;

    println!("Response: {}", response.text);
    println!("Model: {:?}", response.routing_info.model);
    println!("Confidence: {:.2}%", response.confidence * 100.0);

    // Provide feedback for learning
    llm.feedback(Feedback {
        request_id: response.request_id,
        rating: Some(5),
        correction: None,
        task_success: Some(true),
    }).await?;

    Ok(())
}

SIMD Inference Engine

use ruvllm::{SimdInferenceEngine, SimdGenerationConfig, SimdOps};

// Create SIMD-optimized engine
let engine = SimdInferenceEngine::new(256, 128, 4, 4)?;

// Configure generation
let config = SimdGenerationConfig {
    max_tokens: 50,
    temperature: 0.7,
    top_p: 0.9,
    ..Default::default()
};

// Generate with SIMD acceleration
let result = engine.generate("Once upon a time", &config)?;

SONA Learning Loops

use ruvllm::sona::{LoopCoordinator, SonaConfig, InstantLoop, BackgroundLoop};

// Initialize SONA coordinator
let config = SonaConfig {
    hidden_dim: 256,
    embedding_dim: 256,
    pattern_clusters: 100,
    ..Default::default()
};

let coordinator = LoopCoordinator::new(config);

// Instant learning (per-request)
coordinator.instant_loop().record_trajectory(query, response, quality);

// Background learning (hourly)
coordinator.background_loop().extract_patterns().await;

// Deep learning (weekly) - automatically handles EWC++
coordinator.deep_consolidation().await;

Federated Learning

use ruvector_sona::training::{EphemeralAgent, FederatedCoordinator, SonaConfig};

// Create central coordinator (persistent, large capacity)
let mut coordinator = FederatedCoordinator::default_coordinator("main", 3072);
coordinator.set_quality_threshold(0.4);  // Only accept high-quality trajectories
coordinator.set_consolidation_interval(50);  // Auto-consolidate every 50 agents

// Create ephemeral agents for distributed learning
let mut agent = EphemeralAgent::default_federated("agent-1", 3072);

// Agent processes tasks and learns locally
agent.process_trajectory(
    embedding,      // Query embedding
    activations,    // Hidden state activations
    quality,        // Quality score [0.0, 1.0]
    Some("gpt-4".to_string()),  // Model route
    vec!["code".to_string()],   // Context tags
);

// Export state before agent termination
let export = agent.export_state();
println!("Agent exported {} trajectories", export.trajectories.len());

// Coordinator aggregates learning from all agents
let result = coordinator.aggregate(export);
println!("Accepted: {}, Rejected: {}",
    result.trajectories_accepted,
    result.trajectories_rejected
);

// Get patterns for warm-starting new agents
let patterns = coordinator.get_initial_patterns(10);

WASM Usage (Browser/Edge)

Build SONA for WebAssembly:

# Build WASM package
cd crates/sona
wasm-pack build --target web --features wasm

Use in JavaScript:

import init, { WasmSonaEngine } from './pkg/sona.js';

async function main() {
  await init();

  // Create SONA engine
  const engine = new WasmSonaEngine(256);  // hidden_dim = 256

  // Or with custom configuration
  const engineCustom = WasmSonaEngine.withConfig({
    hidden_dim: 256,
    embedding_dim: 256,
    micro_lora_rank: 2,
    base_lora_rank: 16,
    ewc_lambda: 1000.0,
    pattern_clusters: 128,
  });

  // Start trajectory
  const embedding = new Float32Array(256).fill(0.1);
  const trajectoryId = engine.startTrajectory(embedding);

  // Record steps
  engine.recordStep(trajectoryId, 42, 0.8, 1000);

  // End trajectory with quality score
  engine.endTrajectory(trajectoryId, 0.85);

  // Apply LoRA transformation
  const input = new Float32Array(256).fill(1.0);
  const output = engine.applyLora(input);

  // Run learning cycles
  engine.runInstantCycle();  // Flush micro-LoRA updates
  if (engine.tick()) {       // Background learning
    console.log('Background learning completed');
  }

  // Get statistics
  const stats = engine.stats();
  console.log('Patterns:', stats.patterns_stored);
}

HuggingFace Export

Export learned patterns, LoRA weights, and preference pairs to HuggingFace:

# Export LoRA weights in PEFT-compatible SafeTensors format
ruvllm-export safetensors ./exports/lora

# Export learned patterns as JSONL dataset
ruvllm-export patterns ./exports/patterns

# Export DPO/RLHF preference pairs
ruvllm-export preferences ./exports/preferences

# Export all artifacts
ruvllm-export all ./exports

# Push to HuggingFace Hub
HF_TOKEN=your_token ruvllm-export push username/my-sona-model

# Generate pretraining pipeline configuration
ruvllm-export pretrain ./exports

Architecture Deep Dive

HNSW Memory Index

The memory system uses Hierarchical Navigable Small World graphs:

Layer 2:  [3] ─────────────────── [7]
           │                       │
Layer 1:  [3] ─── [5] ─────────── [7] ─── [9]
           │      │                │       │
Layer 0:  [1]─[2]─[3]─[4]─[5]─[6]─[7]─[8]─[9]─[10]

• M = 32 connections per node
• ef_construction = 200 for build quality
• ef_search = 64 for query speed
• O(log N) search complexity

FastGRNN Router

Sparse + Low-rank matrices for efficient routing:

           Input (128-dim)
                │
        ┌───────┴───────┐
        │  LayerNorm    │
        └───────┬───────┘
                │
    ┌───────────┴───────────┐
    │   FastGRNN Cell       │
    │                       │
    │  W_sparse (90% zero)  │
    │  U = A @ B (rank-8)   │
    │                       │
    │  z = σ(Wx + Uh + b)   │
    │  h' = z⊙h + (1-z)⊙ν   │
    └───────────┬───────────┘
                │
        ┌───────┴───────┐
        │ Output Heads  │
        ├───────────────┤
        │ Model Select  │ → 4 classes
        │ Context Size  │ → 5 buckets
        │ Temperature   │ → continuous
        │ Top-p         │ → continuous
        │ Confidence    │ → continuous
        └───────────────┘

MicroLoRA Architecture

Two-tier LoRA system for adaptive learning:

┌─────────────────────────────────────────────────────────────┐
│                      MicroLoRA (Rank 1-2)                   │
│                   Per-Request Adaptation                    │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   Input ──► Down Proj ──► Up Proj ──► Scale ──► Add        │
│   (dim)     (dim→rank)   (rank→dim)   (α/r)    to output   │
│                                                             │
│   Performance: <100μs latency, 2,236 ops/sec               │
│   Rank-2 is ~5% faster than Rank-1 (better SIMD)           │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                      BaseLoRA (Rank 4-16)                   │
│                   Background Adaptation                     │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   Aggregated from successful MicroLoRA patterns             │
│   Merged hourly into base weights                           │
│   EWC++ regularization prevents forgetting                  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

EWC++ (Enhanced Elastic Weight Consolidation)

Prevents catastrophic forgetting:

Loss = Task_Loss + λ * Σᵢ Fᵢ(θᵢ - θ*ᵢ)²

Where:
• Fᵢ = Online Fisher information (EMA decay 0.999)
• θ*ᵢ = Optimal weights for previous tasks
• λ = Adaptive (2000 default, range 100-15000)
• Multi-task memory with circular buffer (10 tasks)
• Automatic task boundary detection

SIMD Operations

Native CPU acceleration:

// AVX2 dot product (8 floats at a time)
#[target_feature(enable = "avx2")]
unsafe fn dot_product_avx2(a: &[f32], b: &[f32]) -> f32

// SSE4.1 fallback (4 floats at a time)
#[target_feature(enable = "sse4.1")]
unsafe fn dot_product_sse(a: &[f32], b: &[f32]) -> f32

// Automatic detection and dispatch
let result = SimdOps::dot_product(&a, &b);

Supported Models

Real Inference (CPU SIMD)

Model Parameters Context Repo
SmolLM 135M 135M 2048 HuggingFaceTB/SmolLM-135M
SmolLM 360M 360M 2048 HuggingFaceTB/SmolLM-360M
Qwen2 0.5B 500M 4096 Qwen/Qwen2-0.5B
TinyLlama 1.1B 1.1B 2048 TinyLlama/TinyLlama-1.1B-Chat

All models support Q4_K_M quantization for efficient CPU inference.

HTTP Server API

When running with the server feature:

Endpoint Method Description
/health GET Health check
/query POST Submit query
/stats GET Get statistics
/feedback POST Submit feedback
/session POST Create new session 
# Example query
curl -X POST http://localhost:3000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What is Rust?", "session_id": null}'

Testing

# Run all tests
cargo test -p ruvllm

# Unit tests only (47 tests)
cargo test -p ruvllm --lib

# Integration tests (15 tests)
cargo test -p ruvllm --test integration

# With output
cargo test -p ruvllm -- --nocapture

Test Coverage

Module Tests Coverage
Memory (HNSW) 12 Search, insertion, graph expansion
Router (FastGRNN) 8 Forward pass, training, EWC
Attention 6 Multi-head, edge features, cross-attention
Embedding 9 Tokenization, caching, pooling
SONA 10 LoRA, EWC++, ReasoningBank, loops
Orchestrator 2 End-to-end pipeline
Integration 15 Full system tests

Project Structure

examples/ruvLLM/
├── Cargo.toml              # Dependencies and features
├── README.md               # This file
├── src/
│   ├── lib.rs              # Library entry point
│   ├── config.rs           # Configuration system
│   ├── error.rs            # Error types
│   ├── types.rs            # Core domain types
│   ├── orchestrator.rs     # Main RuvLLM coordinator
│   ├── memory.rs           # HNSW memory service
│   ├── router.rs           # FastGRNN router
│   ├── attention.rs        # Graph attention engine
│   ├── embedding.rs        # Embedding service
│   ├── inference.rs        # Mock inference pool
│   ├── inference_real.rs   # Candle-based real inference
│   ├── simd_inference.rs   # SIMD-optimized transformer
│   ├── learning.rs         # Self-learning service
│   ├── compression.rs      # Memory compression
│   ├── training.rs         # Pretraining pipeline
│   ├── sona/               # SONA module
│   │   ├── mod.rs          # Module exports
│   │   ├── types.rs        # SONA types
│   │   ├── lora.rs         # MicroLoRA & BaseLoRA
│   │   ├── ewc.rs          # EWC++ implementation
│   │   ├── reasoning_bank.rs  # Pattern storage
│   │   ├── trajectory.rs   # Trajectory recording
│   │   ├── engine.rs       # SONA engine
│   │   └── loops/          # Temporal learning loops
│   │       ├── instant.rs  # Per-request loop
│   │       ├── background.rs  # Hourly loop
│   │       └── coordinator.rs # Loop coordinator
│   └── bin/
│       ├── demo.rs         # Interactive demo
│       ├── bench.rs        # Quick benchmarks
│       ├── benchmark_suite.rs  # Full benchmark suite
│       ├── simd_demo.rs    # SIMD capabilities demo
│       ├── pretrain.rs     # Pretraining pipeline
│       ├── export.rs       # HuggingFace export
│       └── server.rs       # HTTP server
├── tests/
│   └── integration.rs      # Integration tests
├── benches/
│   ├── pipeline.rs         # Full pipeline benchmarks
│   ├── router.rs           # Router benchmarks
│   ├── memory.rs           # Memory benchmarks
│   ├── attention.rs        # Attention benchmarks
│   └── sona_bench.rs       # SONA benchmarks
├── config/                 # Configuration files
└── docs/
    └── sparc/              # SPARC methodology docs

Feature Flags

RuvLLM Features

Feature Default Description
storage ✓ Persistent storage and HNSW indexing
metrics ✓ Prometheus metrics export
server ✗ HTTP server with Axum
real-inference ✗ Candle-based real LLM inference
hf-export ✗ HuggingFace export via ruvector-sona
full ✗ All features enabled 
# Build with all features
cargo build --release --features full

ruvector-sona Features (Dependency)

Feature Default Description
serde-support ✓ Serialization for export, training, and federated learning
wasm ✗ WebAssembly bindings for browser/edge deployment
napi ✗ N-API bindings for Node.js integration 
# Build SONA with WASM support
cd crates/sona
wasm-pack build --target web --features wasm

Configuration Options

Option Default Description
embedding.dimension 768 Embedding vector size
embedding.max_tokens 512 Max tokens per input
memory.hnsw_m 16 HNSW connections per node
memory.hnsw_ef_construction 100 Build quality parameter
memory.hnsw_ef_search 64 Search quality parameter
router.input_dim 128 Router input features
router.hidden_dim 64 FastGRNN hidden size
router.sparsity 0.9 Weight matrix sparsity
router.rank 8 Low-rank decomposition
learning.enabled true Enable self-learning
learning.quality_threshold 0.7 Min quality for writeback
learning.ewc_lambda 2000 EWC regularization strength
sona.pattern_clusters 100 K-means++ clusters
sona.micro_lora_rank 2 MicroLoRA rank

Federated Learning Configuration

Option Default Description
federated.quality_threshold 0.4 Min quality for trajectory acceptance
federated.consolidation_interval 50 Auto-consolidate every N agents
federated.coordinator_capacity 50000 Trajectory buffer size for coordinator
federated.agent_capacity 500 Trajectory buffer size per agent
federated.base_lora_rank 16 Coordinator LoRA rank (deeper for aggregation)

Self-Learning Improvement Over Time

Epoch Queries Quality Routing Cache Hit Memory Improvement
0 0 65.0% 50.0% 0.0% 0 0.0% (baseline)
1 50 67.2% 58.0% 10.0% 25 +3.4%
2 100 69.8% 66.0% 20.0% 50 +7.4%
3 150 71.5% 74.0% 30.0% 75 +10.0%
4 200 73.2% 82.0% 40.0% 100 +12.6%
5 250 74.8% 90.0% 50.0% 125 +15.1%

References

  • LFM2: Liquid Foundation Models - Gated convolutions + grouped query attention
  • FastGRNN - Fast, Accurate, Stable and Tiny GRU
  • HNSW - Hierarchical Navigable Small World Graphs
  • EWC - Elastic Weight Consolidation
  • LoRA - Low-Rank Adaptation of Large Language Models

License

Licensed under either of:

at your option.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Built with Rust + Ruvector
Self-Learning AI that gets smarter with every interaction