eval: 5000-turn long horizon results — pre-built DB regression + federated 100-agent OOM

[rysweet]

5000-Turn Long Horizon Eval — Distributed Hive Mind

Architecture

All agents share a single DistributedHiveGraph backed by a DHT (consistent hash ring). Facts are sharded across agents with replication factor R=3. Queries route to shard owners, not all agents.

graph TB
    subgraph "Single DHT Ring — All N Agents"
        direction LR
        R["Consistent Hash Ring<br/>64 virtual nodes per agent<br/>Facts hashed to ring positions"]
    end

    subgraph "Agent Shards (each holds ~F/N facts)"
        A0["Agent 0<br/>Shard: ~50 facts"]
        A1["Agent 1<br/>Shard: ~48 facts"]
        A2["Agent 2<br/>Shard: ~52 facts"]
        AN["Agent N<br/>Shard: ~50 facts"]
    end

    R --> A0
    R --> A1
    R --> A2
    R --> AN
    A0 <-.->|"bloom filter gossip"| A1
    A1 <-.->|gossip| A2
    A2 <-.->|gossip| AN

Loading

Query Flow

sequenceDiagram
    participant Q as Question
    participant DHT as DHT Router
    participant S1 as Shard Owner 1
    participant S2 as Shard Owner 2
    participant C as Consensus

    Q->>DHT: "What is Sarah Chen birthday?"
    DHT->>DHT: hash key terms → find shard owners
    DHT->>S1: search shard (facts found)
    DHT->>S2: search shard (facts found)
    S1-->>C: "March 15"
    S2-->>C: "March 15"
    C->>Q: Consensus answer

Loading

Learning Flow

sequenceDiagram
    participant T as Turn Pool (5000)
    participant W as ThreadPool (10 workers)
    participant A as Agent (learns 50 turns)
    participant LLM as LLM
    participant DHT as DHT Ring

    T->>W: Round-robin distribute turns
    W->>A: learn batch (parallel)
    A->>LLM: extract facts
    LLM-->>A: structured facts
    A->>A: Store in local Kuzu DB (256MB)
    A->>DHT: promote_fact → replicate to R=3 agents
    Note over W: 10 agents learn simultaneously (9x speedup)

Loading

---

Eval Results

Single Agent Baseline: 94.1%

Level	Score	Questions
L1 direct recall	97.9%	31
L2 multi-source synthesis	100%	5
L3 temporal reasoning	83.3%	3
L4 procedural	100%	2
L5 contradiction	87.5%	2
L6 incremental update	100%	2
L7 teaching	83.3%	2
L8 confidence	89.2%	2
L9 causal	100%	2
L10 counterfactual	70.8%	2
L11 novel skill	100%	1
L12 far transfer	62.5%	2

Runtime: 21.7h (21.6h learning, 84s Q&A+grading). Model: claude-sonnet-4-5-20250929.

Federated Hive Progression

Version	Median	Stddev	Agents	Notes
v1 (naive, longest-wins)	40.0%	—	100	No routing, query all agents
v3 (consensus+routing, broken)	34.9%	31.2%	100	Empty root hive, random fallback
v3 Opus	3.6%	15.5%	100	+ rate limit errors swallowed
Single DHT smoke test	58.8%	4.3%	10	Correct routing, stable
Single DHT full (pending)	TBD	TBD	100	Running now (~3h remaining)

---

Key Bugs Found and Fixed

P0: Empty Root Hive (fixed in PR #18)

Facts stored in per-group hives during learning but queries routed through empty root hive → random agent fallback → 31% stddev. Fix: single DHT ring with all agents.

P0: Kuzu mmap OOM (fixed in PR #11, #2876)

kuzu.Database() defaults to 80% of system RAM + 8TB mmap per DB. 100 agents = crash. Fix: bounded to 256MB per agent.

P1: Sequential Learning (fixed in PR #17)

5000 turns learned one at a time despite 100 agents. Fix: ThreadPoolExecutor with parallel batches → 9x speedup (21.6h → 2.4h).

P1: Swallowed Errors

_synthesize_with_llm() catches all exceptions silently, masking rate limits as "internal error". Opus scored 3.6% because most answers failed with masked 429 errors.

P2: Longest-Answer-Wins (fixed in PR #17)

Querying 100 agents and picking the longest response. Fix: expertise routing via DHT + no-info filtering + Jaccard consensus.

---

PRs

Repo	PR	Status	Change
amplihack-memory-lib	#11	Merged	buffer_pool_size param on CognitiveMemory
amplihack-agent-eval	#17	Merged	DHT, parallel learning, consensus, median-of-3
amplihack-agent-eval	#18	Open	Single DHT ring fix (routing bug)
amplihack	#2876	Open	DistributedHiveGraph, DHT, bloom, docs

Release Assets

Tag	Repo	Contents
dataset-5000t-seed42-v1.0	eval	Pre-built single-agent 5000t Kuzu DB
federated-100agent-5000t-v1.0	eval	100 federated agent Kuzu DBs

Success Criteria (from #2866)

• Single agent >75% — 94.1%
• 100-agent hive ≥ single agent — smoke test 58.8%, full eval pending
• No OOM with 100 DBs — fixed (12.3s, 4.8GB)
• Parallel learning speedup — 9x (21.6h → 2.4h)
• Gossip convergence >90% — pending
• Variance < 10% stddev — 4.3% (was 31.2%)

Related

• eval: 5000-turn long horizon learning test — single agent + 100-agent hive (20 groups × 5) #2866 — Original eval spec
• PR feat: distributed hive mind — federation, LearningAgent eval, retrieval pipeline #2717 — Distributed hive mind implementation
• Architecture docs

---

Next Steps Issues

• #2890 — Task delegation between hive agents
• #2891 — Agent discovery and health monitoring
• #2892 — Persistent storage (NFS-backed Kuzu)
• #2893 — Per-fact embedding routing for 100-agent scale

[rysweet]

Single-Agent Baseline Complete: 94.1%

The full single-agent 5000-turn learning run finished.

Results

Metric	Value
Overall score	94.1%
Runtime	21.7 hours
Learning	21.6h (77,940s)
Q&A + Grading	84s
Questions	56

Per-level breakdown:

Level	Score	Questions
L1 direct recall	97.9%	31
L2 multi-source synthesis	100%	5
L3 temporal reasoning	83.3%	3
L4 procedural	100%	2
L5 contradiction	87.5%	2
L6 incremental update	100%	2
L7 teaching	83.3%	2
L8 confidence	89.2%	2
L9 causal	100%	2
L10 counterfactual	70.8%	2
L11 novel skill	100%	1
L12 far transfer	62.5%	2

Success Criteria Update

• Single agent achieves >75% overall — ✅ 94.1% (exceeds target by 19pp)
• 100-agent hive achieves ≥ single agent — IN PROGRESS (13,372 API calls, ~60% through learning)
• multi_hop_reasoning: hive scores higher — PENDING
• needle_in_haystack: >60% at 5000-turn scale — N/A (not in security scenario categories)
• No OOM or Kuzu crashes with 100 concurrent DBs — ✅ Fixed via DistributedHiveGraph + bounded Kuzu buffer pool
• Federation tree doesn't degrade query latency >5x — PENDING
• Gossip convergence >90% — PENDING

Finding: Federated Learning Is Not Parallelized

The federated 100-agent run is taking the same wall-clock time as single-agent because _run_federated feeds turns sequentially through _MultiAgentAdapter.learn(). Each turn's LLM calls (temporal extraction + fact extraction) happen one at a time. The 100 agents don't learn in parallel — they take turns learning one turn each, sequentially.

To get the expected 20-100x speedup, the learning loop needs to be parallelized (e.g., ThreadPoolExecutor distributing batches of turns to agents concurrently).

PRs

Repo	PR	Status
amplihack-memory-lib	PR #11	CI passing, ready to merge
amplihack-agent-eval	PR #17	CI passing, ready to merge
amplihack	PR #2876	Holding

[rysweet]

Federated 100-Agent Eval Complete: 40.0%

The parallel federated run finished in 2.5 hours (vs 21.7h single-agent). Parallel learning speedup: 9x with 10 workers.

Head-to-Head

Metric	Single Agent	100-Agent Federated
Overall	94.1%	40.0%
Learning time	21.6h	2.4h (9x faster)
Hive facts	—	84,470

Per-Level

Level	Single	Fed	Delta
L1 direct recall	97.9%	29.2%	-68.7pp
L2 multi-source	100%	50.0%	-50pp
L5 contradiction	87.5%	83.3%	-4.2pp
L8 confidence	89.2%	77.5%	-11.7pp
L10 counterfactual	70.8%	70.8%	0pp
L12 far transfer	62.5%	62.5%	0pp

Analysis

The federated hive works mechanically (no OOM, 9x parallel speedup, 84K facts stored). But retrieval quality is poor — L1 recall dropped 68pp.

Root causes:

1. Answer aggregation is naive — "longest answer wins" across 100 agents, not expertise-routed
2. Each agent only sees 50 turns — no gossip rounds run after learning
3. No semantic routing to find the K agents most likely to have the answer

Next Steps

1. Replace "longest answer wins" with expertise-routed answering via DHT query
2. Run gossip rounds after learning
3. Semantic routing: embed question, find K relevant shard owners

Success Criteria

• Single agent >75% — 94.1%
• 100-agent hive >= single — 40.0% (needs retrieval fix)
• No OOM with 100 DBs — Fixed
• Parallel speedup — 9x

[rysweet]

v3 Federated Results: 73.8% (up from 40%)

Fixes: consensus voting, expertise routing via DHT, gossip after learning, no-info filtering.

Metric	Single	v1 (naive)	v3 (fixed)
Overall	94.1%	40.0%	73.8%
Learning	21.6h	2.4h	2.3h
Q&A time	56s	352s	75s (4.7x faster)
Hive facts	—	84K	150K

Biggest wins: L6 incremental 0%->100% (gossip), L1 recall 29%->71% (routing), L11 novel 67%->100%.

Q&A dropped from 352s to 75s by routing to K=10 agents instead of all 100.

Remaining 20pp gap vs single-agent likely from routing not always finding the right shard. More gossip rounds or semantic routing could close it further.

Success Criteria

• Single agent >75% — 94.1%
• 100-agent hive >= single — 73.8% (improved +33.8pp, 20pp remaining gap)
• No OOM — Fixed
• Parallel speedup — 9x
• L6 incremental works — 100% (was 0%)
• L5 contradiction — 100% (hive exceeds single agent!)

[rysweet]

Distributed Hive Mind Architecture

System Overview

graph TB
    subgraph "Single DHT Ring — All 100 Agents"
        direction LR
        R["Consistent Hash Ring<br/>64 virtual nodes per agent<br/>Facts hashed to ring positions"]
    end

    subgraph "Agent Shards (each holds ~F/N facts)"
        A0["Agent 0<br/>Shard: ~50 facts"]
        A1["Agent 1<br/>Shard: ~48 facts"]
        A2["Agent 2<br/>Shard: ~52 facts"]
        AN["Agent 99<br/>Shard: ~50 facts"]
    end

    subgraph "Gossip Layer"
        G["Bloom Filter Exchange<br/>O(log N) convergence"]
    end

    R --> A0
    R --> A1
    R --> A2
    R --> AN
    A0 <-.->|gossip| A1
    A1 <-.->|gossip| A2
    A2 <-.->|gossip| AN

Loading

Query Flow

sequenceDiagram
    participant Q as Question
    participant DHT as DHT Router
    participant S1 as Shard Owner 1
    participant S2 as Shard Owner 2
    participant S3 as Shard Owner 3
    participant C as Consensus

    Q->>DHT: "What is Sarah Chen birthday?"
    DHT->>DHT: hash("sarah chen birthday") → ring position
    DHT->>S1: search shard (3 facts found)
    DHT->>S2: search shard (1 fact found)
    DHT->>S3: search shard (0 facts)
    S1-->>C: "March 15"
    S2-->>C: "March 15"
    C->>Q: Consensus: "March 15" (2/3 agree)

Loading

Learning Flow (Parallel, 10 workers)

sequenceDiagram
    participant T as Turn Pool (5000)
    participant W as ThreadPool (10 workers)
    participant A42 as Agent 42
    participant LLM as LLM (Sonnet)
    participant DHT as Single DHT Ring

    T->>W: Distribute 50 turns to Agent 42
    W->>A42: learn batch
    A42->>LLM: extract_facts("Server prod-db-01...")
    LLM-->>A42: [{content: "prod-db-01 runs PostgreSQL", confidence: 0.9}]
    A42->>A42: Store in local Kuzu DB (256MB)
    A42->>DHT: promote_fact()
    DHT->>DHT: hash → replicate to R=3 agents
    Note over W: 10 agents learn simultaneously

Loading

Design: Single DHT Ring

All 100 agents share one DistributedHiveGraph. Every fact is replicated to R=3 agents via consistent hashing. Queries route to the correct shard owners — no empty root, no random fallback.

graph LR
    S[Single DHT Ring<br/>ALL 100 agents<br/>R=3 replication<br/>Facts sharded across ring]
    style S fill:#6f6

Loading

Performance

Metric	InMemoryHiveGraph	DistributedHiveGraph
100-agent creation	OOM crash (8TB mmap)	12.3s, 4.8GB RSS
Memory per agent	O(F) all facts	O(F/N) shard only
Query fan-out	O(N) all agents	O(K=5) shard owners
Gossip convergence	N/A	O(log N) rounds

Eval Results

Run	Model	Median	Stddev	Notes
Single agent	Sonnet	94.1%	—	Baseline (21.7h)
Smoke test (single DHT, 10 agents)	Sonnet	58.8%	4.3%	Stable routing
Full 5000t (single DHT, 100 agents)	Sonnet	TBD	TBD	Running now

Architecture Docs

Full docs: docs/distributed_hive_mind.md (PR #2876)

[rysweet]

Single-DHT Fix: Smoke Test Results

PR: amplihack-agent-eval#18 — single DHT ring with all agents

Root cause found and fixed: Facts were stored in per-group hives but queries hit the empty root hive, causing fallback to random agents. Fix: one DistributedHiveGraph with all agents on a single DHT ring.

Smoke Test (50 turns, 10 agents, 3 repeats)

Metric	Before (multi-ring, broken)	After (single DHT)
Repeat 1	82.6%	58.8%
Repeat 2	34.9%	57.0%
Repeat 3	23.9%	65.2%
Median	34.9%	58.8%
Stddev	31.2%	4.3%

The stddev dropped 7x (31.2% → 4.3%) — routing now consistently finds the right agents instead of picking randomly.

Full 5000-turn eval in progress

900/5000 turns learned, ETA ~3h. Will post final 3-repeat median results when complete.

[rysweet]

Full 100-Agent Single-DHT Results (3-repeat median)

Metric	Value
Median	47.2%
Stddev	29.6%
Repeats	81.1%, 22.2%, 47.2%
Learning	2.3h (parallel, 10 workers)
Hive facts	273,524

Comparison Across All Runs

Run	Agents	Median	Stddev
Single agent	1	94.1%	—
Smoke (single DHT)	10	58.8%	4.3%
Full (single DHT)	100	47.2%	29.6%

Analysis

Variance at 100 agents (29.6% stddev) is much worse than at 10 agents (4.3%). The 81% run proves facts ARE in the DHT. The 22% run shows keyword-based routing fails to find them consistently at scale.

The DHT routes by hashing key terms from the question — at 100 agents with 64 virtual nodes each (6400 ring positions), small differences in term extraction lead to completely different shard owners per run.

Remaining Issues

1. Routing quality degrades with agent count — keyword hashing is too coarse for 100-agent rings
2. Need semantic routing — embed questions, match against shard content embeddings for stable routing
3. Memory not integrated into OODA loop — design doc at docs/memory_ooda_integration.md (PR feat: distributed hive mind with DHT sharding #2876)

Next Steps

1. Implement semantic routing in DHT (embed question → find similar shard embeddings)
2. Wire Memory into the OODA loop per the design doc
3. Re-run eval with the unified agent

[rysweet]

Semantic Routing + OODA Integration: Smoke Test

Both workstreams completed and pushed to PR #2876:

• caa25d89 — semantic DHT routing via embeddings + Memory wired into OODA loop

Smoke Test Results (50 turns, 10 agents, 3 repeats)

Condition	Median	Stddev	Repeats
Single agent	91.2%	3.3%	90%, 91%, 96%
Federated	65.7%	6.7%	54%, 66%, 66%

Progression

Version	Fed Median	Fed Stddev	What Changed
v1 naive	40.0%	—	Longest-answer-wins
Broken multi-ring (3-rep)	34.9%	31.2%	Empty root hive
Single DHT keyword	58.8%	4.3%	Fixed routing target
Semantic + OODA	65.7%	6.7%	Embedding-based shard routing + memory in OODA loop

What Changed

Semantic routing: DHTRouter now embeds the question and computes cosine similarity against per-shard summary embeddings (running average of all stored fact embeddings). Routes to the K shards with highest similarity instead of keyword hash lookup.

OODA integration: AgenticLoop now has observe() + orient() phases that call memory.remember() and memory.recall(). LearningAgent.learn_from_content() and answer_question() route through the OODA loop instead of bypassing it. The agent being evaluated is now the real goal-seeking agent, not a simplified pipeline.

[rysweet]

Full 100-Agent Eval: Semantic Routing + OODA (Latest Code)

Condition	Median	Stddev	Repeats	Time
Federated (100 agents)	45.8%	21.7%	82.8%, 44.5%, 45.8%	2.7h
Single agent	Pending	—	—	Running

Progression (all 100-agent, 5000-turn, 3-repeat)

Version	Median	Stddev	Key change
Broken multi-ring	34.9%	31.2%	Empty root hive
Single DHT keyword	47.2%	29.6%	Fixed routing target
Semantic + OODA	45.8%	21.7%	Embedding routing + OODA loop

Analysis

Semantic routing did not improve at 100-agent scale. The shard summary embedding (running average of all stored fact embeddings) becomes too blurry with 50 diverse facts per shard — it cannot distinguish "Sarah Chen birthday" from "PostgreSQL port" because both are averaged together.

The 82.8% first repeat proves the knowledge IS there. The 44-45% on repeats 2-3 shows the routing is inconsistent.

What would actually help

The retrieval bottleneck is not the topology — it is the routing precision. Options:

1. Per-fact embeddings instead of shard averages — route by finding the K most similar individual facts across all shards
2. Inverted index — keyword → shard mapping maintained during learning, queried at recall time
3. Reduce agent count — 10-20 agents with 250-500 facts each route much better than 100 agents with 50 facts each

[rysweet]

100 Agents Deployed to Azure

20 Container Apps running in Azure, 5 agents each = 100 agents total.

All connected to Azure Service Bus for inter-agent communication.

$ az containerapp list --resource-group hive-mind-rg --query "length(@)"
20

$ az containerapp list --query "[].{name:name, running:properties.runningStatus}" -o table
Name              Running
amplihive-app-0   Running
amplihive-app-1   Running
...
amplihive-app-19  Running

Agent logs show clean startup:

Starting agent: name=agent-0 topology=hive transport=azure_service_bus backend=simple
Agent agent-0 memory initialized and entering OODA loop
Agent agent-0 stats (tick=20): {backend: simple, topology: single, fact_count: 1}

Infrastructure

• Region: westus2
• Container Apps: 20 (5 agents each)
• Service Bus: hive-sb-dj2qo2w7vu5zi
• ACR: hivacrhivemind
• Backend: simple (in-memory, Kuzu not compatible with Azure Files SMB locks)

Known Limitations

• Memory is in-memory (not persistent across restarts) — need NFS-backed Azure Files or Managed Disks for Kuzu persistence
• Agents use simple topology (not distributed DHT) within each container — need to wire NetworkGraphStore to Service Bus for cross-container gossip
• No learning content fed yet — agents are running OODA loops but have no domain knowledge

Next Steps

1. Wire NetworkGraphStore to Service Bus for cross-container distributed memory
2. Feed learning content to agents and verify cross-agent recall
3. Run the 5000-turn eval across the distributed hive
4. Add Kuzu persistence via NFS or Managed Disks

[rysweet]

Cross-Container Memory Sharing Active

All agents now running with topology: distributed via NetworkGraphStore + Azure Service Bus.

Agent agent-0  (app-0):  topology=distributed, transport=azure_service_bus
Agent agent-25 (app-5):  topology=distributed, transport=azure_service_bus  
Agent agent-50 (app-10): topology=distributed, transport=azure_service_bus
...

Architecture (Deployed)

Container App 0 (5 agents)     Container App 1 (5 agents)     ... Container App 19
├── agent-0                    ├── agent-5                         ├── agent-95
├── agent-1                    ├── agent-6                         ├── agent-96
├── agent-2                    ├── agent-7                         ├── agent-97
├── agent-3                    ├── agent-8                         ├── agent-98
└── agent-4                    └── agent-9                         └── agent-99
     │                              │                                    │
     └──────────── Azure Service Bus (hive-sb-dj2qo2w7vu5zi) ───────────┘
                   Writes → publish to topic
                   Reads → subscribe + merge local results

Commits on PR #2876

• 834aeba6 — Wire NetworkGraphStore for distributed topology via Azure Service Bus
• 27582b9c — Use simple memory backend for containers
• 0e708ed6 — Fix Kuzu directory creation
• Plus 15 earlier commits (DHT, bloom, GraphStore, Memory facade, OODA, docs, etc.)

What Works

• 100 agents running across 20 Azure Container Apps
• Distributed topology with Service Bus transport
• OODA loop running on each agent
• Memory sharing via publish/subscribe pattern

Remaining

• Feed learning content and verify cross-agent recall end-to-end
• Fix routing precision for better eval scores
• Add Kuzu persistence (NFS or Managed Disks)
• Run distributed eval across Azure hive