chore: Remove obsolete documentation files and update references

- Remove docs/guides/ (replaced by Diátaxis _tutorials/_howto/_reference/_explanation)
- Remove docs/README.md (referenced removed Scaladoc API)
- Remove release-notes/ (consolidated into CHANGELOG.md)
- Remove DATAFRAME_API_EXAMPLES.md (covered by tutorials)
- Remove RELEASE_NOTES_0.6.0.md (consolidated into CHANGELOG.md)
- Remove HACKERNEWS_ANNOUNCEMENT.md (promotional material)
- Update AGENTS.md to reference new Diátaxis structure
- Update CHANGELOG.md with documentation restructure and removed files
- Update ROADMAP.md decision log with documentation changes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Author: derrickburns

AGENTS.md

@@ -3,9 +3,9 @@
 Use this guide to make concise, high-signal contributions to the generalized k-means clustering library.
 
 ## Project Structure & Module Organization
-- Scala sources live in `src/main/scala` (DataFrame/ML API under `com.massivedatascience.clusterer.ml`), with version-specific shims in `src/main/scala-2.12` and `src/main/scala-2.13`. Legacy RDD code remains in `com.massivedatascience.clusterer`.
+- Scala sources live in `src/main/scala` (DataFrame/ML API under `com.massivedatascience.clusterer.ml`), with version-specific shims in `src/main/scala-2.12` and `src/main/scala-2.13`.
 - Tests use ScalaTest under `src/test/scala` with Spark-local fixtures; shared data is in `src/test/resources`. Executable examples sit in `src/main/scala/examples`.
-- Python wrapper lives in `python/` (`massivedatascience` package, examples, and tests). Docs and release notes are in `docs/`, `release-notes/`, `ARCHITECTURE.md`, and `DATAFRAME_API_EXAMPLES.md`.
+- Python wrapper lives in `python/` (`massivedatascience` package, examples, and tests). Documentation follows Diátaxis structure in `docs/` (`_tutorials/`, `_howto/`, `_reference/`, `_explanation/`). Architecture and changelog are in `ARCHITECTURE.md` and `CHANGELOG.md`.
 
 ## Build, Test, and Development Commands
 - `sbt compile` — compile against the default Scala/Spark matrix; use `sbt ++2.13.14` or `sbt ++2.12.18` to pin versions.

CHANGELOG.md

@@ -9,7 +9,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 - Comprehensive CI validation DAG with cross-version testing
-- Production quality blockers documented in ACTION_ITEMS.md
 - SECURITY.md with vulnerability reporting guidelines
 - CONTRIBUTING.md with development guidelines
 - Test suite fixes for Scala 2.12/2.13 compatibility
@@ -63,11 +62,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - Inertia (WCSS) computation
   - Cluster sizes and balance metrics
   - Elbow curve helper for finding optimal k
-- **Documentation guides** in `docs/guides/`:
-  - Quick Start Guide - get running in 5 minutes
-  - Divergence Selection Guide - comprehensive decision flowchart and examples
-  - X-Means Auto-K Demo - automatic cluster count selection with BIC/AIC
-  - Soft Clustering Guide - interpreting probabilistic memberships
+- **Documentation restructured** using Diátaxis framework:
+  - `docs/_tutorials/` - Learning-oriented guides (first clustering, PySpark, algorithm selection)
+  - `docs/_howto/` - Task-oriented guides (installation, choosing divergence, finding optimal k, outliers)
+  - `docs/_reference/` - Information-oriented (parameters, algorithms, divergences)
+  - `docs/_explanation/` - Understanding-oriented (Bregman divergences, Lloyd's algorithm, performance)
+  - Custom domain: https://generalized-kmeans-clustering.massivedatascience.com
+  - Jekyll-based site with automatic deployment
+- **SPECIFICATION.md** - Compressed specification enabling AI reconstruction of the codebase
 - **Test suites for new components** (215 new tests, 854 total):
   - OutlierDetectionSuite: 16 tests for distance-based and trimmed outlier detection
   - SparseBregmanKernelSuite: 28 tests for sparse-optimized SE, KL, L1, Spherical kernels
@@ -106,6 +108,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Removed
 - Legacy RDD API and associated coreset/transform modules (DataFrame/ML API is now the sole surface)
+- Obsolete documentation files consolidated into Diátaxis structure:
+  - `ACTION_ITEMS.md` (superseded by ROADMAP.md)
+  - `DATAFRAME_API_EXAMPLES.md` (covered by tutorials)
+  - `RELEASE_NOTES_0.6.0.md` (consolidated into CHANGELOG.md)
+  - `HACKERNEWS_ANNOUNCEMENT.md` (promotional material)
+  - `docs/guides/` (replaced by Diátaxis categories)
+  - `release-notes/` directory (consolidated into CHANGELOG.md)
 
 ## [0.6.0] - 2025-10-18