feat(pipeline): enforce JSON schemas for all stage artifacts

docs: mark P1.3 complete and queue final schema follow-ups

Author: KristjanESPERANTO

package-lock.json

@@ -50,6 +50,8 @@
     ]
   },
   "dependencies": {
+    "ajv": "^8.17.1",
+    "ajv-formats": "^3.0.1",
     "marked": "^16.3.0",
     "normalize-package-data": "^8.0.0",
     "npm-check-updates": "^18.3.0",

package.json

@@ -17,8 +17,10 @@ This document captures the long-term improvements we want to implement in the mo
 | ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | ------ |
 | P1.1 | Describe the full stage graph (inputs/outputs, side-effects) in a machine-readable config ([stage graph](pipeline/stage-graph.json)) ✅ Completed Sep 2025 | none         | S      |
 | P1.2 | Introduce a lightweight orchestrator (Node CLI) that reads the config and runs stages with structured logging                                              | P1.1         | M      |
-| P1.3 | Add JSON-schema validation for every stage boundary (modules.stage.\* files)                                                                               | P1.1         | M      |
+| P1.3 | Add JSON-schema validation for every stage boundary (modules.stage.\* files) ([schemas](pipeline/schemas)) ✅ Completed Sep 2025                           | P1.1         | M      |
 | P1.4 | Provide a skip/only mechanism for partial runs (e.g. `--only=checks`)                                                                                      | P1.2         | S      |
+| P1.5 | Define schemas for published outputs (`modules.json`, `modules.min.json`, `stats.json`) and enforce validation in release packaging                        | P1.3         | M      |
+| P1.6 | Consolidate shared schema definitions (shared `$defs` / generator) to keep stage contracts in sync                                                         | P1.3         | S      |
 
 ### 2. Runtime & Codebase Consolidation
 
@@ -74,15 +76,16 @@ These topics sit adjacent to the pipeline work but should stay visible while pri
 
 ## Execution Strategy
 
-1. **Foundation first**: Complete P1.3 and P5.1 to ensure we fully understand the existing pipeline and keep stage contracts safe while refactoring.
+1. **Foundation first**: Focus on P5.1 to visualize the current architecture now that P1.3 is in place.
 2. **Consolidate utilities** (P2.1) before migrating Python scripts so we can share code and avoid duplicated logic.
 3. **Migrate in slices**: Move `get_modules` first (lower risk) and keep Python fallbacks until the TS version is proven stable. Follow with `check_modules` in feature flags (`--checks=legacy|ts`).
 4. **Add tests alongside migrations** to prevent regressions and make future refactors safer.
 5. **Keep communication tight** via roadmap review meetings or async updates in GitHub Discussions.
 
 ## Next Concrete Steps
 
-1. Create issues for P1.3 and P5.1 and assign initial owners.
-2. Prepare a small sample dataset to validate the staging files while we work on schema validation.
+1. Create an issue for P5.1 and assign an initial owner.
+2. Prepare a small sample dataset to validate the staging files against the new schemas.
+3. File follow-up issues for P1.5 and P1.6 (final artifact schemas + shared definitions) and slot them after P5.1.
 
 Feel free to adjust priorities, rename tasks, or add new items. This roadmap is meant to stay alive—update it as soon as we learn something new during implementation.

pipeline-refactor-roadmap.md

@@ -0,0 +1,72 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "https://magicmirror.builders/schemas/modules.stage.1.json",
+  "title": "Modules Stage 1",
+  "type": "object",
+  "required": ["lastUpdate", "modules"],
+  "properties": {
+    "lastUpdate": {
+      "type": "string",
+      "format": "date-time"
+    },
+    "modules": {
+      "type": "array",
+      "items": {
+        "$ref": "#/$defs/module"
+      }
+    }
+  },
+  "additionalProperties": false,
+  "$defs": {
+    "module": {
+      "type": "object",
+      "required": [
+        "name",
+        "category",
+        "url",
+        "id",
+        "maintainer",
+        "description",
+        "issues"
+      ],
+      "properties": {
+        "name": {
+          "type": "string",
+          "minLength": 1
+        },
+        "category": {
+          "type": "string",
+          "minLength": 1
+        },
+        "url": {
+          "type": "string",
+          "format": "uri"
+        },
+        "id": {
+          "type": "string",
+          "minLength": 1
+        },
+        "maintainer": {
+          "type": "string",
+          "minLength": 1
+        },
+        "maintainerURL": {
+          "type": "string"
+        },
+        "description": {
+          "type": "string"
+        },
+        "issues": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        },
+        "outdated": {
+          "type": "string"
+        }
+      },
+      "additionalProperties": true
+    }
+  }
+}

pipeline/schemas/modules.stage.1.schema.json

@@ -0,0 +1,74 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "https://magicmirror.builders/schemas/modules.stage.2.json",
+  "title": "Modules Stage 2",
+  "type": "array",
+  "items": {
+    "$ref": "#/$defs/module"
+  },
+  "$defs": {
+    "module": {
+      "type": "object",
+      "required": [
+        "name",
+        "category",
+        "url",
+        "id",
+        "maintainer",
+        "description",
+        "issues"
+      ],
+      "properties": {
+        "name": {
+          "type": "string",
+          "minLength": 1
+        },
+        "category": {
+          "type": "string",
+          "minLength": 1
+        },
+        "url": {
+          "type": "string",
+          "format": "uri"
+        },
+        "id": {
+          "type": "string",
+          "minLength": 1
+        },
+        "maintainer": {
+          "type": "string",
+          "minLength": 1
+        },
+        "maintainerURL": {
+          "type": "string"
+        },
+        "description": {
+          "type": "string"
+        },
+        "issues": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        },
+        "outdated": {
+          "type": "string"
+        },
+        "stars": {
+          "type": "integer",
+          "minimum": 0
+        },
+        "license": {
+          "type": "string"
+        },
+        "hasGithubIssues": {
+          "type": "boolean"
+        },
+        "isArchived": {
+          "type": "boolean"
+        }
+      },
+      "additionalProperties": true
+    }
+  }
+}