Skip to content

Commit 0c7be6b

Browse files
KristjanESPERANTOKristjanESPERANTO
committed
docs: update README and add pipeline modernization roadmap
1 parent 3682246 commit 0c7be6b

File tree

2 files changed

+92
-35
lines changed

2 files changed

+92
-35
lines changed

‎README.md‎

Lines changed: 3 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -123,41 +123,9 @@ How can you use it?
123123

124124
## Ideas / To do
125125

126-
- Long-term goal: The website functions like an app store. The user only has to click on an install button and the module is installed and a basic configuration is inserted.
127-
- Move ideas/todos to GitHub Issues and add proper tags.
128-
- `package.json` is handled in different scripts. This could be done in one.
129-
- Show results in popup on the page.
130-
- Compatibility with:
131-
- the module list of `MMM-Remote-Control`: <https://github.com/Jopyth/MMM-Remote-Control/blob/master/modules.json> / `MMM-Remote-Control-Repository` <https://github.com/MMRIZE/MMM-Remote-Control-Repository>
132-
- and mmpm <https://github.com/Bee-Mar/mmpm>
133-
- optimize progress information while running `node --run all`
134-
- Also run the tests on MagicMirror² core?
135-
- Tests
136-
- Is repository reachable? Now the get_modules script interrupts if a repo isn't reachable.
137-
- If a module isn't marked as outdated, but has no commits since years: Check if the module is set to archived on GitHub/GitLab.
138-
- Is depandabot there?
139-
- Is it set to monthly?
140-
- Only production?
141-
- rewrite the scripts in rust (it could be faster and a good practice)
142-
- extend the package.json in modules to collect more additional information (based on a [proposal from MMRIZE](https://forum.magicmirror.builders/topic/18092/automatic-checking-of-all-magicmirror-modules/45?_=1702858630364))
143-
144-
```json
145-
"MagicMirror": {
146-
"screenshot": {
147-
"license": "MIT",
148-
"url": "https://somewhere.com/screenshot.png"
149-
},
150-
"required": {
151-
"mm": "2.25",
152-
"node": "18.0",
153-
"ApiKey": true
154-
},
155-
"notice" : [
156-
"This will not work in Windows.",
157-
"Pre-dependency required. Please read README."
158-
]
159-
}
160-
```
126+
See [GitHub issues](https://github.com/MagicMirrorOrg/MagicMirror-3rd-Party-Modules/issues) for ideas and tasks.
127+
128+
Track pipeline modernization progress in [`pipeline-refactor-roadmap.md`](pipeline-refactor-roadmap.md).
161129

162130
### Templates for the creation of issues
163131

‎pipeline-refactor-roadmap.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Pipeline Modernization Roadmap
2+
3+
This document captures the long-term improvements we want to implement in the module processing pipeline. It converts the recent analysis into actionable workstreams so that we can tackle them step by step.
4+
5+
## Guiding Objectives
6+
7+
- **Reduce maintenance friction** by consolidating runtimes, centralizing utilities, and improving observability.
8+
- **Keep the pipeline resilient** through schema validation, caching, and clearer failure handling.
9+
- **Improve contributor experience** with better documentation, configurability, and faster feedback loops.
10+
- **Preserve current functionality** while iterating in small, testable increments.
11+
12+
## Workstreams
13+
14+
### 1. Pipeline Architecture & Orchestration
15+
16+
| Task | Description | Dependencies | Effort |
17+
| ---- | ------------------------------------------------------------------------------------------------------------- | ------------ | ------ |
18+
| P1.1 | Describe the full stage graph (inputs/outputs, side-effects) in a machine-readable config | none | S |
19+
| P1.2 | Introduce a lightweight orchestrator (Node CLI) that reads the config and runs stages with structured logging | P1.1 | M |
20+
| P1.3 | Add JSON-schema validation for every stage boundary (modules.stage.\* files) | P1.1 | M |
21+
| P1.4 | Provide a skip/only mechanism for partial runs (e.g. `--only=checks`) | P1.2 | S |
22+
23+
### 2. Runtime & Codebase Consolidation
24+
25+
| Task | Description | Dependencies | Effort |
26+
| ---- | ----------------------------------------------------------------------------------------------- | ------------ | ------ |
27+
| P2.1 | Extract shared utilities (HTTP, Git, FS, logging, rate-limiter) into a reusable Node/TS module | none | M |
28+
| P2.2 | Port `get_modules.py` to TypeScript, reusing the shared utilities | P2.1 | L |
29+
| P2.3 | Port `check_modules.py` logic incrementally (start with fast checks, then optional heavy tasks) | P2.1 | XL |
30+
| P2.4 | Enable TypeScript build tooling (tsconfig, lint) and cover new modules with tests | P2.1 | M |
31+
| P2.5 | Centralize `package.json` ingestion so data is parsed once and shared across stages | P2.1 | M |
32+
33+
### 3. Robustness & Performance Safety Nets
34+
35+
| Task | Description | Dependencies | Effort |
36+
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | ------ |
37+
| P3.1 | Add persistent caches for API responses and HEAD validations with expiration metadata | none | M |
38+
| P3.2 | Introduce a central rate limiter + retry strategy for GitHub/GitLab requests | P3.1 | M |
39+
| P3.3 | Capture structured logs (JSON) and aggregate per-stage timing metrics | P1.2 | M |
40+
| P3.4 | Ensure deterministic outputs (sorted keys, hash-based image names) and document the guarantees | P1.2 | S |
41+
| P3.5 | Harden repository clone flow to gracefully skip missing/renamed repos and keep the pipeline green ([#41](https://github.com/MagicMirrorOrg/MagicMirror-3rd-Party-Modules/issues/41)) | none | M |
42+
| P3.6 | Replace hard-coded star fallbacks with authenticated API lookups for non-GitHub hosts ([#5](https://github.com/MagicMirrorOrg/MagicMirror-3rd-Party-Modules/issues/5)) | P3.1 | M |
43+
44+
### 4. Checks & Developer Experience
45+
46+
| Task | Description | Dependencies | Effort |
47+
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | ------ |
48+
| P4.1 | Split checks into a registry with metadata (category, severity, auto-fixable) | P2.3 | M |
49+
| P4.2 | Add configuration file to toggle check groups (`fast`, `deep`, optional ESLint/ncu) | P4.1 | S |
50+
| P4.3 | Create sample dataset + regression tests for check outputs (golden files) | P4.1 | M |
51+
| P4.4 | Provide CLI progress UI and Markdown summary per run | P1.2 | S |
52+
| P4.5 | Add rule detecting modules that rely on MagicMirror core dependencies without declaring them ([#78](https://github.com/MagicMirrorOrg/MagicMirror-3rd-Party-Modules/issues/78)) | P4.1 | M |
53+
| P4.6 | Check README install/update sections for copyable fenced command blocks ([#54](https://github.com/MagicMirrorOrg/MagicMirror-3rd-Party-Modules/issues/54)) | P4.1 | S |
54+
| P4.7 | Recommend `npm install --omit=dev` when modules expose devDependencies in instructions ([#53](https://github.com/MagicMirrorOrg/MagicMirror-3rd-Party-Modules/issues/53)) | P4.1 | S |
55+
| P4.8 | Flag modules with multi-year inactivity that are not marked `outdated` and nudge maintainers to review status | P4.1 | M |
56+
| P4.9 | Inspect Dependabot configs for schedule scope (monthly cadence, production-only) and suggest adjustments | P4.1 | M |
57+
58+
### 5. Documentation & Collaboration
59+
60+
| Task | Description | Dependencies | Effort |
61+
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------- | ------ |
62+
| P5.1 | Publish an architecture diagram (current + target) in `docs/` | P1.1 | S |
63+
| P5.2 | Expand contributor guide with setup, pipeline tips, troubleshooting | P5.1 | M |
64+
| P5.3 | Convert roadmap tasks into GitHub issues and track via project board | after roadmap review | S |
65+
| P5.4 | Schedule periodic checkpoint (e.g. monthly) to review progress & adjust priorities | P5.3 | S |
66+
| P5.5 | Coordinate migration of selected `guides/` content into the official docs repository ([#59](https://github.com/MagicMirrorOrg/MagicMirror-3rd-Party-Modules/issues/59)) | P5.2 | M |
67+
| P5.6 | Establish change protocol with wiki-dependent consumers (e.g., MMPM) before altering layout ([#40](https://github.com/MagicMirrorOrg/MagicMirror-3rd-Party-Modules/issues/40)) | P5.1 | S |
68+
69+
### Related Backlog Items
70+
71+
These topics sit adjacent to the pipeline work but should stay visible while prioritising future sprints:
72+
73+
- Progressive loading / chunked delivery of the public `modules.json` to keep client payload manageable ([#8](https://github.com/MagicMirrorOrg/MagicMirror-3rd-Party-Modules/issues/8)).
74+
75+
## Execution Strategy
76+
77+
1. **Foundation first**: Complete P1.1, P1.3, and P5.1 to ensure we fully understand the existing pipeline and keep stage contracts safe while refactoring.
78+
2. **Consolidate utilities** (P2.1) before migrating Python scripts so we can share code and avoid duplicated logic.
79+
3. **Migrate in slices**: Move `get_modules` first (lower risk) and keep Python fallbacks until the TS version is proven stable. Follow with `check_modules` in feature flags (`--checks=legacy|ts`).
80+
4. **Add tests alongside migrations** to prevent regressions and make future refactors safer.
81+
5. **Keep communication tight** via roadmap review meetings or async updates in GitHub Discussions.
82+
83+
## Next Concrete Steps
84+
85+
1. Finalize + merge this roadmap document.
86+
2. Create issues for P1.1, P1.3, P5.1 and assign initial owners.
87+
3. Prepare a small sample dataset to validate the staging files while we work on schema validation.
88+
89+
Feel free to adjust priorities, rename tasks, or add new items. This roadmap is meant to stay alive—update it as soon as we learn something new during implementation.

0 commit comments

Comments
 (0)