Docs overall cleanup; fixes #555

[andrewmusselman]

Documentation Cleanup and Reorganization

Related Issue

Fixes #555

Type of Change

• Documentation update

Summary

Comprehensive cleanup and simplification of the /docs directory. Consolidates redundant content, fills in stub files with real content, and creates a clear navigation structure with a main README table of contents.

Changes

Before: 40 files, ~10,100 lines
After: 25 files, ~5,800 lines (43% reduction)

What Changed

Area	Before	After	Notes
Root	No main README	README.md with TOC	Central navigation hub
database-service/	12 files (~4,500 lines)	database/ 4 files (~750 lines)	Consolidated, removed DBA-level detail
testing/	9 files (~1,000 lines)	7 files (~2,400 lines)	Reorganized, added comprehensive guides
developers/	2 files	2 files (kept)	Now linked from main README
Stubs	Empty placeholders	Real content	architecture.md, roadmap.md

New Files

• README.md - Main documentation hub with quick links table and full TOC
• architecture.md - System overview with ASCII diagram, component descriptions
• data-store.md - Agent data store documentation (new feature)
• testing/backend-testing.md - Comprehensive Python/pytest guide
• testing/frontend-testing.md - Comprehensive React/Vitest guide
• testing/integration-testing.md - Backend integration + Playwright E2E
• testing/ci-cd.md - GitHub Actions workflows and coverage

Removed/Consolidated

Database docs consolidated:

• database-service/architecture.md → merged into database/README.md
• database-service/interface.md → merged into database/README.md
• database-service/implementing-new-database.md → merged into database/README.md
• database-service/performance.md → removed (enterprise-level detail)
• database-service/security.md → removed (enterprise-level detail)
• database-service/migration.md → brief note in database/troubleshooting.md
• database-service/testing.md → removed (covered in main testing docs)
• database-service/implementations/ (4 files) → removed (links to external docs)

Testing docs reorganized:

• testing/overview.md → merged into testing/README.md
• testing/coverage.md → merged into testing/README.md and testing/ci-cd.md
• testing/SUMMARY.md → removed (redundant)

Stubs removed:

• license.md → points to repo root LICENSE
• contact-community.md → content in main README "Getting Help"
• database-service.md → was just a redirect

Files Kept As-Is

• quickstart/README.md (with images)
• developers-quickstart.md
• llm-provider-configuration.md
• api-key-management.md
• api.md
• observability.md
• EXTENSION_EXAMPLE.md
• about-name-origin.md
• developers/llm-service.md
• developers/website.md
• database/schema.md
• testing/unit-testing.md
• testing/contributing.md

New Structure

docs/
├── README.md                      # NEW: Main TOC
├── quickstart/README.md           # Kept
├── architecture.md                # NEW: Real content
├── roadmap.md                     # Updated: Links to wiki
├── data-store.md                  # NEW: Agent data store
│
├── database/                      # REORGANIZED (was database-service/)
│   ├── README.md                  # Consolidated overview
│   ├── configuration.md           # Kept
│   ├── schema.md                  # Kept  
│   └── troubleshooting.md         # Consolidated
│
├── testing/                       # REORGANIZED
│   ├── README.md                  # Overview + quick reference
│   ├── unit-testing.md            # Kept
│   ├── backend-testing.md         # NEW: Comprehensive
│   ├── frontend-testing.md        # NEW: Comprehensive
│   ├── integration-testing.md     # NEW: Backend + E2E
│   ├── ci-cd.md                   # NEW: GitHub Actions
│   └── contributing.md            # Kept
│
├── developers/                    # Kept
│   ├── llm-service.md
│   └── website.md
│
├── developers-quickstart.md       # Kept
├── llm-provider-configuration.md  # Kept
├── api-key-management.md          # Kept
├── api.md                         # Kept
├── observability.md               # Kept
├── EXTENSION_EXAMPLE.md           # Kept
└── about-name-origin.md           # Kept

Migration Notes

Broken Links to Fix

If any external links point to old paths:

Old Path	New Path
database-service/README.md	database/README.md
database-service/configuration.md	database/configuration.md
database-service/schema.md	database/schema.md
database-service/troubleshooting.md	database/troubleshooting.md
database-service/* (other files)	database/README.md or removed
testing/overview.md	testing/README.md
testing/coverage.md	testing/README.md
testing/backend-testing.md	testing/backend-testing.md (same, but expanded)
testing/frontend-testing.md	testing/frontend-testing.md (same, but expanded)

Docusaurus Sync

The website/scripts/sync-docs.js may need updates if it references specific file paths. The sync should handle the new structure automatically since it processes whatever is in /docs.

Related Issues

• Addresses documentation cleanup and organization
• Adds documentation for agent data store feature ([Feature]: Add persistent data layer #550)

---

By submitting this PR, I confirm that I have read and agree to follow the project's Code of Conduct and Contributing Guidelines.

[rawkintrevo]

I like some of the adds, but the deletions were really slack and burn and a bit much for me.

[andrewmusselman]

I like some of the adds, but the deletions were really slack and burn and a bit much for me.

Yes I know what you mean; anything in particular you want put back in?

[andrewmusselman]

Will pull back in old database-service and testing files