Directory restructure + RAG token budget + persona recovery (#271)

* Relocate source from src/debug/jtag/ to src/ + root cleanup

Move all application source code up from the historical 3-level-deep
src/debug/jtag/ path to src/. Delete 155 unused root files (old configs,
debug dumps, screenshots, dead scripts). Update all hardcoded path
references across CI workflows, precommit hook, CLAUDE.md, README.md,
and ~70 documentation files.

Verified: TypeScript compiles, 923 Rust tests pass, npm start deploys
with 254 commands + 17 daemons, ping + chat working.

* Token budget enforcement: replace guessed message counts with actual token accumulation

ConversationHistorySource was converting its allocated token budget to a message
count using TOKENS_PER_MESSAGE_ESTIMATE=50, then fetching exactly that many messages
and returning ALL of them without verifying total tokens. With 200+ token messages
this caused 4-5x budget overruns and context window overflows on DeepSeek, Fireworks,
Together, and other providers.

Fix: fetch generous batch (500), convert to LLM format, iterate newest-to-oldest
accumulating actual token counts (chars/3), stop when allocated budget is exhausted.
Token budget is now the ONLY constraint — no artificial message caps.

ChatRAGBuilder: removed Math.min(50) hard cap that prevented 128K models from using
more than 50 messages. Generous fetch limit now scales with context window.

* Fix remaining src/debug/jtag path references across docs, scripts, and configs

* Consolidate overflow conversation history instead of silently dropping it

Old messages that exceed the token budget are now compressed to
"SenderName: first line..." and prepended as a conversation summary.
85% of budget goes to recent verbatim messages, 15% reserved for
consolidated older messages. The AI sees the full conversation arc
instead of losing everything beyond the cutoff.

* Fix persona stuck on failed messages: always advance bookmark

When evaluateAndPossiblyRespondWithCognition threw (API 400, timeout,
network error), the bookmark was never updated. Rust's tick loop
re-polled the same un-bookmarked message, re-enqueued it, and the
persona retried the same failed message forever — silently stuck.

Wrap response in try/finally so the bookmark always advances.
A failed response attempt must not block the entire queue.

Author: joelteply

.continuum/genome/SETUP-COMPLETE.md

@@ -30,7 +30,7 @@
 ### 2. TypeScript Integration
 
 **Modified Files:**
-- `src/debug/jtag/system/genome/fine-tuning/server/adapters/UnslothLoRAAdapter.ts`
+- `src/system/genome/fine-tuning/server/adapters/UnslothLoRAAdapter.ts`
   - `supportsFineTuning()`: Checks if environment bootstrapped
   - `executeUnslothTraining()`: Uses wrapper script (auto-activates conda)
   - Clear error messages if environment missing
@@ -299,7 +299,7 @@ If Unsloth import fails, training falls back to standard PyTorch (slower but wor
 
 .gitignore                     # MODIFIED - Added genome paths
 
-src/debug/jtag/system/genome/fine-tuning/server/adapters/
+src/system/genome/fine-tuning/server/adapters/
 └── UnslothLoRAAdapter.ts     # MODIFIED - Uses wrapper script
 ```
 
@@ -319,7 +319,7 @@ src/debug/jtag/system/genome/fine-tuning/server/adapters/
 ## Questions?
 
 **Setup issues:** See `.continuum/genome/python/README.md`
-**Architecture questions:** See `src/debug/jtag/system/user/server/modules/PERSONA-CONVERGENCE-ROADMAP.md`
+**Architecture questions:** See `src/system/user/server/modules/PERSONA-CONVERGENCE-ROADMAP.md`
 **Training errors:** Run test script first to verify environment
 
 **Philosophy:** "Test the shit out of it" - Every piece is validated before integration.

.continuum/genome/python/README.md

@@ -101,7 +101,7 @@ python3 -c "import unsloth; print(unsloth.__version__)"
 
 ```bash
 .continuum/genome/python/train-wrapper.sh \
-  src/debug/jtag/system/genome/fine-tuning/server/adapters/scripts/peft-train.py \
+  src/system/genome/fine-tuning/server/adapters/scripts/peft-train.py \
   --config config.json \
   --output output/
 ```
@@ -245,5 +245,5 @@ If training fails with unclear errors:
 4. Check logs in temporary test directory
 
 For architecture questions, see:
-- `src/debug/jtag/system/user/server/modules/PERSONA-CONVERGENCE-ROADMAP.md`
-- `src/debug/jtag/system/user/server/modules/LORA-GENOME-PAGING.md`
+- `src/system/user/server/modules/PERSONA-CONVERGENCE-ROADMAP.md`
+- `src/system/user/server/modules/LORA-GENOME-PAGING.md`

.continuum/genome/python/test-training.sh

@@ -75,7 +75,7 @@ echo ""
 mkdir -p "$TEST_DIR/output"
 
 if "$SCRIPT_DIR/train-wrapper.sh" \
-    "$(dirname "$SCRIPT_DIR")/../../src/debug/jtag/system/genome/fine-tuning/server/adapters/scripts/unsloth-train.py" \
+    "$(dirname "$SCRIPT_DIR")/../../src/system/genome/fine-tuning/server/adapters/scripts/unsloth-train.py" \
     --config "$TEST_DIR/config.json" \
     --output "$TEST_DIR/output"; then
 

.eslintignore

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     defaults:
       run:
-        working-directory: src/debug/jtag
+        working-directory: src
 
     steps:
     - uses: actions/checkout@v4
@@ -21,7 +21,7 @@ jobs:
       with:
         node-version: '20'
         cache: 'npm'
-        cache-dependency-path: src/debug/jtag/package-lock.json
+        cache-dependency-path: src/package-lock.json
 
     - name: Install dependencies
       run: npm ci

.eslintrc.js

@@ -4,13 +4,13 @@ on:
   push:
     branches: [ main ]
     paths:
-      - 'src/debug/jtag/package.json'
-      - 'src/debug/jtag/package-lock.json'
+      - 'src/package.json'
+      - 'src/package-lock.json'
   pull_request:
     branches: [ main ]
     paths:
-      - 'src/debug/jtag/package.json'
-      - 'src/debug/jtag/package-lock.json'
+      - 'src/package.json'
+      - 'src/package-lock.json'
   schedule:
     - cron: '0 0 * * 0'  # Run weekly on Sunday at midnight
 
@@ -19,7 +19,7 @@ jobs:
     runs-on: ubuntu-latest
     defaults:
       run:
-        working-directory: src/debug/jtag
+        working-directory: src
 
     steps:
     - uses: actions/checkout@v4
@@ -29,7 +29,7 @@ jobs:
       with:
         node-version: '20'
         cache: 'npm'
-        cache-dependency-path: src/debug/jtag/package-lock.json
+        cache-dependency-path: src/package-lock.json
 
     - name: Install dependencies
       run: npm ci

.github/workflows/ci.yml

@@ -3,14 +3,14 @@ name: Validate Continuum Config
 on:
   pull_request:
     paths:
-      - 'src/debug/jtag/**'
+      - 'src/**'
 
 jobs:
   validate:
     runs-on: ubuntu-latest
     defaults:
       run:
-        working-directory: src/debug/jtag
+        working-directory: src
 
     steps:
     - uses: actions/checkout@v4
@@ -20,7 +20,7 @@ jobs:
       with:
         node-version: '20'
         cache: 'npm'
-        cache-dependency-path: src/debug/jtag/package-lock.json
+        cache-dependency-path: src/package-lock.json
 
     - name: Install dependencies
       run: npm ci

.github/workflows/dependencies.yml

@@ -13,7 +13,7 @@ build/
 continuum-jtag-*.tgz
 
 # Generated manifest files (use generated.ts instead)
-src/debug/jtag/manifests/
+src/manifests/
 
 # Generated test files
 test-config.md
@@ -162,7 +162,7 @@ src/ui/**/*.d.ts
 src/commands/**/*.js
 src/commands/**/*.js.map
 src/commands/**/*.d.ts
-/src/debug/jtag/.archive
+/src/.archive
 .tgz
 
 # Runtime directories (session data, logs, temp files)
@@ -178,4 +178,4 @@ src/commands/**/*.d.ts
 !CLAUDE.md
 !test*output.txt
 _archive/
-src/debug/jtag/.continuum/sessions/validation/
+src/.continuum/sessions/validation/

.github/workflows/validate-continuum.yml

@@ -337,7 +337,7 @@ let results = algo.execute(&input);
 4. **Repeat**
 
 ```bash
-cd src/debug/jtag
+cd src
 npm start                    # DEPLOYS code changes, takes 130s or so
 
 ./jtag ping #check for server and browser connection
@@ -970,7 +970,7 @@ npm start
 
 ### Documentation
 
-**Full Architecture**: `src/debug/jtag/system/user/server/modules/`
+**Full Architecture**: `src/system/user/server/modules/`
 - `AUTONOMOUS-LOOP-ROADMAP.md` - RTOS-inspired servicing
 - `SELF-MANAGED-QUEUE-DESIGN.md` - AI autonomy through tasks
 - `LORA-GENOME-PAGING.md` - Virtual memory for skills
@@ -1144,7 +1144,7 @@ The AIs will:
 **Fix**: Always take screenshot after deployment
 
 ### 3. WRONG WORKING DIRECTORY
-**Always work from**: `src/debug/jtag`
+**Always work from**: `src`
 **Commands**: `./jtag` NOT `./continuum`
 
 ### 4. IGNORE EXISTING TYPES
@@ -1433,20 +1433,20 @@ grep -r "UserEntity\|ChatMessageEntity" daemons/data-daemon/ | grep -v EntityReg
 # Should return zero results (except EntityRegistry.ts)
 ```
 
-### **[UNIVERSAL-PRIMITIVES.md](src/debug/jtag/docs/UNIVERSAL-PRIMITIVES.md)**
+### **[UNIVERSAL-PRIMITIVES.md](src/docs/UNIVERSAL-PRIMITIVES.md)**
 Commands.execute() and Events.subscribe()/emit() - the two primitives everything is built on.
 
-### **[GENERATOR-OOP-PHILOSOPHY.md](src/debug/jtag/docs/GENERATOR-OOP-PHILOSOPHY.md)** - CORE PHILOSOPHY
+### **[GENERATOR-OOP-PHILOSOPHY.md](src/docs/GENERATOR-OOP-PHILOSOPHY.md)** - CORE PHILOSOPHY
 Generators and OOP are intertwined parallel forces:
 - Generators ensure structural correctness at creation time
 - OOP/type system ensures behavioral correctness at runtime
 - AIs should strive to create generators for any repeatable pattern
 - This enables tree-based delegation of ability with compounding capability
 
 ### **PersonaUser Convergence Docs**
-- `src/debug/jtag/system/user/server/modules/PERSONA-CONVERGENCE-ROADMAP.md`
-- `src/debug/jtag/system/user/server/modules/AUTONOMOUS-LOOP-ROADMAP.md`
-- `src/debug/jtag/system/user/server/modules/LORA-GENOME-PAGING.md`
+- `src/system/user/server/modules/PERSONA-CONVERGENCE-ROADMAP.md`
+- `src/system/user/server/modules/AUTONOMOUS-LOOP-ROADMAP.md`
+- `src/system/user/server/modules/LORA-GENOME-PAGING.md`
 
 **Quick tip**: If you're about to write code that duplicates patterns or violates architecture rules, STOP and read ARCHITECTURE-RULES.md first. Then apply the aggressive refactoring principle from this guide.