Skip to content

Commit 4c1c234

Browse files
chore: retire legacy python tooling
1 parent ea63627 commit 4c1c234

File tree

19 files changed

+80
-877
lines changed

19 files changed

+80
-877
lines changed

.gitignore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@
33
/modules_/*
44
/modules_temp/*
55
/node_modules/*
6-
/scripts/__pycache__/*
76
.eslintcache
87
.idea
98
.vscode/*

.pylintrc

Lines changed: 0 additions & 4 deletions
This file was deleted.

container/Dockerfile

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
FROM nikolaik/python-nodejs:latest AS builder
1+
FROM node:slim AS builder
2+
3+
RUN apt-get update \
4+
&& apt-get install -y --no-install-recommends git \
5+
&& rm -rf /var/lib/apt/lists/*
26

37
WORKDIR /workspace
48

@@ -10,7 +14,6 @@ set -e
1014
git config --global --add safe.directory /workspace
1115
git log -1
1216
export GITHUB_TOKEN="${GITHUB_TOKEN}"
13-
pip install requests
1417
npm clean-install
1518
node --run all
1619
EOF

cspell.config.json

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
"bergfex",
1212
"bugsounet",
1313
"checkjs",
14-
"checkpy",
14+
"checkts",
1515
"codeberg",
1616
"Cutover",
1717
"diffing",
@@ -36,7 +36,6 @@
3636
"omxplayer",
3737
"openweathermap",
3838
"ptrbld",
39-
"PYTHONPATH",
4039
"ratp",
4140
"refspecs",
4241
"rollup",

docs/CONTRIBUTING.md

Lines changed: 3 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,8 @@ Welcome! This document bundles the day-to-day tasks contributors perform when wo
66

77
```bash
88
npm install
9-
python3 -m venv .venv && source .venv/bin/activate # optional but recommended for Python scripts
109
```
1110

12-
The Node and Python scripts live side by side today. The roadmap tracks our plan to consolidate runtimes, but for now please ensure both toolchains are available.
13-
1411
## Running the pipeline
1512

1613
You can run each stage individually or rely on the helper commands registered in `package.json`:
@@ -22,7 +19,7 @@ You can run each stage individually or rely on the helper commands registered in
2219
| `node --run getModules` | Stage 3 – clone the repos locally for deeper inspection. |
2320
| `node --run expandModuleList` | Stage 4 – parse `package.json`, pick screenshots, and compute derived metadata. |
2421
| `node --run checkModulesJs` | Stage 5a – JavaScript checks against the cloned repositories. |
25-
| `node --run checkModules` | Stage 5b – Python checks to surface README and packaging issues. |
22+
| `node --run checkModules` | Stage 5b – Deep TypeScript checks to surface README and packaging issues. |
2623
| `node --run all` | Convenience wrapper that executes the full chain in order. |
2724

2825
Heavy stages (`getModules`, `checkModules`) download hundreds of repositories and can take more than 10 minutes. When iterating on faster tasks (Stage 1–2, Stage 4), feel free to skip the expensive steps.
@@ -55,9 +52,7 @@ Runs JavaScript-based checks (naming conventions, minified files, etc.) against
5552

5653
#### Stage 5b – `check-modules/index.ts`
5754

58-
Runs the deep repository analysis in TypeScript, mirroring the legacy Python behavior while sharing utilities with the rest of the pipeline. It parses README files, inspects packaging hygiene, shells out to ESLint/`npm-check-updates`, and produces the markdown summary alongside `modules.json`, `modules.min.json`, and `stats.json`.
59-
60-
> Legacy fallback: the historical Python implementation (`scripts/check_modules.py`) remains available for parity comparisons. Invoke it via `pipeline run --only=check-modules --checks=legacy` when you need to diff outputs against the TypeScript stage.
55+
Runs the deep repository analysis in TypeScript, parsing README files, inspecting packaging hygiene, shelling out to ESLint/`npm-check-updates`, and producing the markdown summary alongside `modules.json`, `modules.min.json`, and `stats.json`.
6156

6257
#### `validate_release_artifacts.js`
6358

@@ -117,7 +112,7 @@ The bundled files live in `dist/schemas/`. They ship with the repository so `nod
117112

118113
## Prerequisites & installation
119114

120-
1. Install [Node.js](https://nodejs.org). Python is optional unless you plan to run the legacy comparison harness.
115+
1. Install [Node.js](https://nodejs.org).
121116
2. Clone the repository:
122117
```bash
123118
git clone https://github.com/MagicMirrorOrg/MagicMirror-3rd-Party-Modules

docs/architecture.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
Visibility into the automation that builds and publishes the third-party module catalogue helps contributors reason about changes and spot failure points early. This document summarizes the current pipeline, highlights the target architecture we are steering toward, and links each element back to the modernization roadmap.
44

5-
## Current state (September 2025)
5+
## Current state (October 2025)
66

7-
The production pipeline is orchestrated via `node scripts/orchestrator/index.js run full-refresh` (or the shorthand npm scripts) and progresses through six sequential stages. Most stages are implemented in TypeScript/Node.js and reuse the shared utility layer introduced in P2.1; the final deep-analysis stage remains Python for now. Each stage produces a well-defined artifact that ships with a JSON Schema contract enforced at the boundary.
7+
The production pipeline is orchestrated via `node scripts/orchestrator/index.js run full-refresh` (or the shorthand npm scripts) and progresses through six sequential stages. All stages are now implemented in TypeScript/Node.js and reuse the shared utility layer introduced in P2.1. Each stage produces a well-defined artifact that ships with a JSON Schema contract enforced at the boundary.
88

99
### Stage overview
1010

@@ -15,19 +15,19 @@ The production pipeline is orchestrated via `node scripts/orchestrator/index.js
1515
| 3 | `get-modules` | TypeScript | `website/data/modules.stage.3.json`, `modules/`, `modules_temp/` |
1616
| 4 | `expand-module-list` | Node.js | `website/data/modules.stage.4.json`, `website/images/` |
1717
| 5 | `check-modules-js` | Node.js | `website/data/modules.stage.5.json` |
18-
| 6 | `check-modules` | Python | `website/data/modules.json`, `website/data/modules.min.json`, `website/data/stats.json`, `website/result.md` |
18+
| 6 | `check-modules` | TypeScript | `website/data/modules.json`, `website/data/modules.min.json`, `website/data/stats.json`, `website/result.md` |
1919

2020
### Current workflow diagram
2121

2222
```mermaid
2323
flowchart LR
2424
orchestrator[["Node orchestrator CLI (stage-graph.json)"]]
25-
orchestrator --> create{{"Create module list - TypeScript"}}
26-
orchestrator --> update{{"Update repository metadata - TypeScript"}}
25+
orchestrator --> create{{"Create module list - Node.js"}}
26+
orchestrator --> update{{"Update repository metadata - Node.js"}}
2727
orchestrator --> fetch{{"Fetch module repos - TypeScript"}}
28-
orchestrator --> enrich{{"Enrich with package metadata - TypeScript"}}
28+
orchestrator --> enrich{{"Enrich with package metadata - Node.js"}}
2929
orchestrator --> checkjs{{"Static checks - Node.js"}}
30-
orchestrator --> checkpy{{"Deep analysis - Python"}}
30+
orchestrator --> checkts{{"Deep analysis - TypeScript"}}
3131
3232
create --> stage1["modules.stage.1.json"]
3333
stage1 --> update
@@ -37,14 +37,14 @@ flowchart LR
3737
fetch --> clones[("modules/, modules_temp/")]
3838
enrich -- "modules.stage.4.json" --> checkjs
3939
enrich --> images[("website/images/")]
40-
checkjs -- "modules.stage.5.json" --> checkpy
41-
checkpy --> outputs[("modules.json, modules.min.json, stats.json, result.md")]
40+
checkjs -- "modules.stage.5.json" --> checkts
41+
checkts --> outputs[("modules.json, modules.min.json, stats.json, result.md")]
4242
```
4343

4444
### Observations
4545

4646
- Stage contracts are codified via the bundled schemas stored under `dist/schemas/` (sources live in `pipeline/schemas/src/`).
47-
- Cross-cutting utilities (HTTP, Git, filesystem, rate limiting) now live in `scripts/shared/` and are reused by the TypeScript stages.
47+
- Cross-cutting utilities (HTTP, Git, filesystem, rate limiting) now live in `scripts/shared/` and are reused by every TypeScript stage, including the deep-analysis step.
4848
- The orchestrator CLI runs the declarative stage graph and supports `--only/--skip`, retries, and shared logging.
4949

5050
### Legacy workflow snapshot (pre-September 2025)
@@ -60,8 +60,8 @@ flowchart LR
6060
getLegacy --> clonesLegacy[("modules/, modules_temp/")]
6161
expandLegacy -- "modules.stage.4.json" --> checkjsLegacy{{"Static checks - Node.js"}}
6262
expandLegacy --> imagesLegacy[("website/images/")]
63-
checkjsLegacy -- "modules.stage.5.json" --> checkpyLegacy{{"Deep analysis - Python"}}
64-
checkpyLegacy --> outputsLegacy[("modules.json, modules.min.json, stats.json, result.md")]
63+
checkjsLegacy -- "modules.stage.5.json" --> checkLegacy{{"Deep analysis - Python"}}
64+
checkLegacy --> outputsLegacy[("modules.json, modules.min.json, stats.json, result.md")]
6565
```
6666

6767
This legacy diagram captures the pre-orchestrator, mixed-runtime pipeline that relied on direct node and Python scripts. Retaining it here provides a historical comparison as we continue to modernize the remaining stages.

docs/pipeline-refactor-roadmap.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,13 @@ This document captures the long-term improvements we want to implement in the mo
2424

2525
### 2. Runtime & Codebase Consolidation
2626

27-
| Task | Description | Dependencies | Effort |
28-
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | ------ |
29-
| P2.1 | Extract shared utilities (HTTP, Git, FS, logging, rate-limiter) into a reusable Node/TS module ✅ Completed Sep 2025 | none | M |
30-
| P2.2 | Port `get_modules.py` to TypeScript, reusing the shared utilities ✅ Completed Sep 2025 (Python fallback removed) | P2.1 | L |
31-
| P2.3 | Port `check_modules.py` logic incrementally (start with fast checks, then optional heavy tasks) ✅ Completed Oct 2025 (TS stage default, Python fallback kept via `--checks=legacy`) | P2.1 | XL |
32-
| P2.4 | Enable TypeScript build tooling (tsconfig, lint) and cover new modules with tests | P2.1 | M |
33-
| P2.5 | Centralize `package.json` ingestion so data is parsed once and shared across stages | P2.1 | M |
27+
| Task | Description | Dependencies | Effort |
28+
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | ------ |
29+
| P2.1 | Extract shared utilities (HTTP, Git, FS, logging, rate-limiter) into a reusable Node/TS module ✅ Completed Sep 2025 | none | M |
30+
| P2.2 | Port `get_modules.py` to TypeScript, reusing the shared utilities ✅ Completed Sep 2025 (Python fallback removed) | P2.1 | L |
31+
| P2.3 | Port `check_modules.py` logic incrementally (start with fast checks, then optional heavy tasks) ✅ Completed Oct 2025 (TS stage now fully TypeScript; Python fallback removed) | P2.1 | XL |
32+
| P2.4 | Enable TypeScript build tooling (tsconfig, lint) and cover new modules with tests | P2.1 | M |
33+
| P2.5 | Centralize `package.json` ingestion so data is parsed once and shared across stages | P2.1 | M |
3434

3535
### 3. Robustness & Performance Safety Nets
3636

@@ -98,6 +98,7 @@ Immediate action items:
9898
1. Kick off P2.4: land TypeScript build/lint tooling across the new stages and wire them into CI.
9999
2. Expand the comparison harness diff coverage (README/HTML, stats thresholds) per the fixture plan follow-ups.
100100
3. Prioritize P4.x registry metadata work now that parity is secured (define categories/severities for existing checks).
101+
4. Decide on the long-term diff strategy (e.g. golden artifacts) now that the Python fallback is gone.
101102

102103
---
103104

docs/pipeline/check-modules-reference.md

Lines changed: 14 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@ This page consolidates the material that previously lived in the P2.3 rollout do
66

77
## Status snapshot
88

9-
- ✅ TypeScript implementation is the default Stage 5 runner; the Python fallback remains available behind `--checks=legacy` for comparison runs only.
10-
- ✅ Comparison harness (`npm run checkModules:compare`) executes both implementations, produces diff artifacts, and runs in CI via `check-modules-compare.yaml`.
11-
- 🔄 Follow-ups tracked here: extend harness diff coverage (README/HTML artifacts) and define warning/failure thresholds before we retire the legacy stage entirely.
9+
- ✅ TypeScript implementation is the default Stage 5 runner.
10+
- ✅ Comparison harness (`npm run checkModules:compare`) can execute multiple commands, capture artifacts, and (when two runs complete) produce diffs for analysis.
11+
- 🔄 Follow-ups tracked here: extend harness diff coverage (README/HTML artifacts) and define warning/failure thresholds ahead of diff gating in CI.
1212

1313
## Rule inventory
1414

@@ -79,8 +79,8 @@ These are the rule IDs currently implemented by the TypeScript checker. Keep thi
7979
- `detect-jquery-local-copy`: warns when a local `jquery.js` copy is bundled.
8080
- `detect-missing-update-section`: ensures README files include `## Update`.
8181
- `detect-missing-install-section`: ensures README files include `## Install`.
82-
- `deprecation_check`: runs `npm-deprecated-check` and aggregates results.
83-
- `eslint_checks`: executes ESLint with the curated configuration to produce findings.
82+
- `dependency-deprecation-helper`: runs `npm-deprecated-check` and aggregates results.
83+
- `eslint-helper`: executes ESLint with the curated configuration to produce findings.
8484

8585
## Fixture coverage
8686

@@ -103,8 +103,8 @@ The curated fixture repositories live in `fixtures/modules/`. Keep this table in
103103
| `synthetic-lockfile` | Synthetic | Legacy lockfile versions | `lock-deprecated-v1`, `lock-deprecated-v2` | Ready | Located under `fixtures/modules/synthetic-lockfile`. |
104104
| `synthetic-readme-heuristics` | Synthetic | README structure + node_modules detection | `detect-missing-install-section`, `detect-missing-update-section`, `detect-node_modules-dir` | Ready | Located under `fixtures/modules/synthetic-readme-heuristics`. |
105105
| `synthetic-jquery-local` | Synthetic | Local jQuery copy | `detect-jquery-local-copy` | Ready | Located under `fixtures/modules/synthetic-jquery-local`. |
106-
| `synthetic-eslint-helper` | Synthetic | ESLint integration | `eslint_checks` | Ready | Located under `fixtures/modules/synthetic-eslint-helper`. |
107-
| `synthetic-deprecation-helper` | Synthetic | npm deprecated packages via helper | `deprecation_check` | Ready | Located under `fixtures/modules/synthetic-deprecation-helper`. |
106+
| `synthetic-eslint-helper` | Synthetic | ESLint integration | `eslint-helper` | Ready | Located under `fixtures/modules/synthetic-eslint-helper`. |
107+
| `synthetic-deprecation-helper` | Synthetic | npm deprecated packages via helper | `dependency-deprecation-helper` | Ready | Located under `fixtures/modules/synthetic-deprecation-helper`. |
108108

109109
### Coverage matrix
110110

@@ -131,23 +131,23 @@ The comparison harness lives under `scripts/check-modules/compare/` and is exerc
131131

132132
### Goals
133133

134-
- Execute legacy (`--checks=legacy`) and TypeScript implementations against the same curated dataset.
135-
- Normalize and diff Stage 5 outputs, surfacing rule-level differences and aggregate stats.
134+
- Execute one or two configured commands against the curated dataset.
135+
- Normalize and diff Stage 5 outputs, surfacing rule-level differences and aggregate stats when two runs are available.
136136
- Emit machine-readable (`diff.json`) and Markdown (`diff.md`) reports for CI artifacts.
137137

138138
### Flow
139139

140140
1. Prepare a temporary workspace with the curated fixtures.
141-
2. Run the legacy stage via the orchestrator, capturing Stage 5 outputs.
142-
3. Run the TypeScript stage (default) in a separate directory.
141+
2. Run the first configured command (labelled `legacy` by default) and capture Stage 5 outputs.
142+
3. Run the second command (`ts` by default) if provided.
143143
4. Normalize artifacts (sorted keys, trimmed timestamps).
144-
5. Run diff logic to produce JSON + Markdown summaries.
144+
5. Run diff logic to produce JSON + Markdown summaries when two runs succeed.
145145
6. Exit non-zero when differences are detected or an execution step fails.
146146

147147
### Implementation notes
148148

149149
- CLI entry point: `node scripts/check-modules/compare/index.js`.
150-
- Supports overrides via `--fixtures`, `--legacy`, `--ts`, and `--output` flags.
150+
- Supports overrides via `--fixtures`, `--legacy`, `--ts`, and `--output` flags (the `--legacy` command defaults to `skip`).
151151
- Uses shared logging utilities; artifacts are collated under the run directory with a `plan.json` descriptor.
152152
- Snapshot-based tests guard the harness itself.
153153

@@ -156,6 +156,7 @@ The comparison harness lives under `scripts/check-modules/compare/` and is exerc
156156
- Add README/HTML artifact comparisons to the diff output.
157157
- Define thresholds that downgrade certain differences to warnings instead of hard failures.
158158
- Consider partial rule subsets to speed up ad-hoc debugging runs.
159+
- Explore storing golden artifacts now that the Python fallback has been retired.
159160

160161
## Housekeeping
161162

0 commit comments

Comments
 (0)