You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- drop python helper scripts in favor of the TS stage 5
- update docs/fixtures to reference the new helpers
- base the builder image on node:latest-slim without pip setup
Copy file name to clipboardExpand all lines: docs/CONTRIBUTING.md
+3-8Lines changed: 3 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,11 +6,8 @@ Welcome! This document bundles the day-to-day tasks contributors perform when wo
6
6
7
7
```bash
8
8
npm install
9
-
python3 -m venv .venv &&source .venv/bin/activate # optional but recommended for Python scripts
10
9
```
11
10
12
-
The Node and Python scripts live side by side today. The roadmap tracks our plan to consolidate runtimes, but for now please ensure both toolchains are available.
13
-
14
11
## Running the pipeline
15
12
16
13
You can run each stage individually or rely on the helper commands registered in `package.json`:
@@ -22,7 +19,7 @@ You can run each stage individually or rely on the helper commands registered in
22
19
|`node --run getModules`| Stage 3 – clone the repos locally for deeper inspection. |
|`node --run checkModulesJs`| Stage 5a – JavaScript checks against the cloned repositories. |
25
-
|`node --run checkModules`| Stage 5b – Python checks to surface README and packaging issues.|
22
+
|`node --run checkModules`| Stage 5b – Deep TypeScript checks to surface README and packaging issues. |
26
23
|`node --run all`| Convenience wrapper that executes the full chain in order. |
27
24
28
25
Heavy stages (`getModules`, `checkModules`) download hundreds of repositories and can take more than 10 minutes. When iterating on faster tasks (Stage 1–2, Stage 4), feel free to skip the expensive steps.
Runs the deep repository analysis in TypeScript, mirroring the legacy Python behavior while sharing utilities with the rest of the pipeline. It parses README files, inspects packaging hygiene, shells out to ESLint/`npm-check-updates`, and produces the markdown summary alongside `modules.json`, `modules.min.json`, and `stats.json`.
59
-
60
-
> Legacy fallback: the historical Python implementation (`scripts/check_modules.py`) remains available for parity comparisons. Invoke it via `pipeline run --only=check-modules --checks=legacy` when you need to diff outputs against the TypeScript stage.
55
+
Runs the deep repository analysis in TypeScript, parsing README files, inspecting packaging hygiene, shelling out to ESLint/`npm-check-updates`, and producing the markdown summary alongside `modules.json`, `modules.min.json`, and `stats.json`.
61
56
62
57
#### `validate_release_artifacts.js`
63
58
@@ -117,7 +112,7 @@ The bundled files live in `dist/schemas/`. They ship with the repository so `nod
117
112
118
113
## Prerequisites & installation
119
114
120
-
1. Install [Node.js](https://nodejs.org). Python is optional unless you plan to run the legacy comparison harness.
Copy file name to clipboardExpand all lines: docs/architecture.md
+12-12Lines changed: 12 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,9 +2,9 @@
2
2
3
3
Visibility into the automation that builds and publishes the third-party module catalogue helps contributors reason about changes and spot failure points early. This document summarizes the current pipeline, highlights the target architecture we are steering toward, and links each element back to the modernization roadmap.
4
4
5
-
## Current state (September 2025)
5
+
## Current state (October 2025)
6
6
7
-
The production pipeline is orchestrated via `node scripts/orchestrator/index.js run full-refresh` (or the shorthand npm scripts) and progresses through six sequential stages. Most stages are implemented in TypeScript/Node.js and reuse the shared utility layer introduced in P2.1; the final deep-analysis stage remains Python for now. Each stage produces a well-defined artifact that ships with a JSON Schema contract enforced at the boundary.
7
+
The production pipeline is orchestrated via `node scripts/orchestrator/index.js run full-refresh` (or the shorthand npm scripts) and progresses through six sequential stages. All stages are now implemented in TypeScript/Node.js and reuse the shared utility layer introduced in P2.1. Each stage produces a well-defined artifact that ships with a JSON Schema contract enforced at the boundary.
8
8
9
9
### Stage overview
10
10
@@ -15,19 +15,19 @@ The production pipeline is orchestrated via `node scripts/orchestrator/index.js
- Stage contracts are codified via the bundled schemas stored under `dist/schemas/` (sources live in `pipeline/schemas/src/`).
47
-
- Cross-cutting utilities (HTTP, Git, filesystem, rate limiting) now live in `scripts/shared/` and are reused by the TypeScript stages.
47
+
- Cross-cutting utilities (HTTP, Git, filesystem, rate limiting) now live in `scripts/shared/` and are reused by every TypeScript stage, including the deep-analysis step.
48
48
- The orchestrator CLI runs the declarative stage graph and supports `--only/--skip`, retries, and shared logging.
This legacy diagram captures the pre-orchestrator, mixed-runtime pipeline that relied on direct node and Python scripts. Retaining it here provides a historical comparison as we continue to modernize the remaining stages.
| P2.1 | Extract shared utilities (HTTP, Git, FS, logging, rate-limiter) into a reusable Node/TS module ✅ Completed Sep 2025 | none | M |
30
-
| P2.2 | Port `get_modules.py` to TypeScript, reusing the shared utilities ✅ Completed Sep 2025 (Python fallback removed) | P2.1 | L |
31
-
| P2.3 | Port `check_modules.py` logic incrementally (start with fast checks, then optional heavy tasks) ✅ Completed Oct 2025 (TS stage default, Python fallback kept via `--checks=legacy`) | P2.1 | XL |
32
-
| P2.4 | Enable TypeScript build tooling (tsconfig, lint) and cover new modules with tests | P2.1 | M |
33
-
| P2.5 | Centralize `package.json` ingestion so data is parsed once and shared across stages | P2.1 | M |
| P2.1 | Extract shared utilities (HTTP, Git, FS, logging, rate-limiter) into a reusable Node/TS module ✅ Completed Sep 2025 | none | M |
30
+
| P2.2 | Port `get_modules.py` to TypeScript, reusing the shared utilities ✅ Completed Sep 2025 (Python fallback removed) | P2.1 | L |
31
+
| P2.3 | Port `check_modules.py` logic incrementally (start with fast checks, then optional heavy tasks) ✅ Completed Oct 2025 (TS stage now fully TypeScript; Python fallback removed) | P2.1 | XL |
32
+
| P2.4 | Enable TypeScript build tooling (tsconfig, lint) and cover new modules with tests | P2.1 | M |
33
+
| P2.5 | Centralize `package.json` ingestion so data is parsed once and shared across stages | P2.1 | M |
34
34
35
35
### 3. Robustness & Performance Safety Nets
36
36
@@ -98,6 +98,7 @@ Immediate action items:
98
98
1. Kick off P2.4: land TypeScript build/lint tooling across the new stages and wire them into CI.
99
99
2. Expand the comparison harness diff coverage (README/HTML, stats thresholds) per the fixture plan follow-ups.
100
100
3. Prioritize P4.x registry metadata work now that parity is secured (define categories/severities for existing checks).
101
+
4. Decide on the long-term diff strategy (e.g. golden artifacts) now that the Python fallback is gone.
Copy file name to clipboardExpand all lines: docs/pipeline/check-modules-reference.md
+14-13Lines changed: 14 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,9 +6,9 @@ This page consolidates the material that previously lived in the P2.3 rollout do
6
6
7
7
## Status snapshot
8
8
9
-
- ✅ TypeScript implementation is the default Stage 5 runner; the Python fallback remains available behind `--checks=legacy` for comparison runs only.
10
-
- ✅ Comparison harness (`npm run checkModules:compare`) executes both implementations, produces diff artifacts, and runs in CI via `check-modules-compare.yaml`.
11
-
- 🔄 Follow-ups tracked here: extend harness diff coverage (README/HTML artifacts) and define warning/failure thresholds before we retire the legacy stage entirely.
9
+
- ✅ TypeScript implementation is the default Stage 5 runner.
10
+
- ✅ Comparison harness (`npm run checkModules:compare`) can execute multiple commands, capture artifacts, and (when two runs complete) produce diffs for analysis.
11
+
- 🔄 Follow-ups tracked here: extend harness diff coverage (README/HTML artifacts) and define warning/failure thresholds ahead of diff gating in CI.
12
12
13
13
## Rule inventory
14
14
@@ -79,8 +79,8 @@ These are the rule IDs currently implemented by the TypeScript checker. Keep thi
79
79
-`detect-jquery-local-copy`: warns when a local `jquery.js` copy is bundled.
80
80
-`detect-missing-update-section`: ensures README files include `## Update`.
81
81
-`detect-missing-install-section`: ensures README files include `## Install`.
82
-
-`deprecation_check`: runs `npm-deprecated-check` and aggregates results.
83
-
-`eslint_checks`: executes ESLint with the curated configuration to produce findings.
82
+
-`dependency-deprecation-helper`: runs `npm-deprecated-check` and aggregates results.
83
+
-`eslint-helper`: executes ESLint with the curated configuration to produce findings.
84
84
85
85
## Fixture coverage
86
86
@@ -103,8 +103,8 @@ The curated fixture repositories live in `fixtures/modules/`. Keep this table in
103
103
|`synthetic-lockfile`| Synthetic | Legacy lockfile versions |`lock-deprecated-v1`, `lock-deprecated-v2`| Ready | Located under `fixtures/modules/synthetic-lockfile`. |
104
104
|`synthetic-readme-heuristics`| Synthetic | README structure + node_modules detection |`detect-missing-install-section`, `detect-missing-update-section`, `detect-node_modules-dir`| Ready | Located under `fixtures/modules/synthetic-readme-heuristics`. |
105
105
|`synthetic-jquery-local`| Synthetic | Local jQuery copy |`detect-jquery-local-copy`| Ready | Located under `fixtures/modules/synthetic-jquery-local`. |
106
-
|`synthetic-eslint-helper`| Synthetic | ESLint integration |`eslint_checks`| Ready | Located under `fixtures/modules/synthetic-eslint-helper`. |
107
-
|`synthetic-deprecation-helper`| Synthetic | npm deprecated packages via helper |`deprecation_check`| Ready | Located under `fixtures/modules/synthetic-deprecation-helper`. |
106
+
|`synthetic-eslint-helper`| Synthetic | ESLint integration |`eslint-helper`| Ready | Located under `fixtures/modules/synthetic-eslint-helper`. |
107
+
|`synthetic-deprecation-helper`| Synthetic | npm deprecated packages via helper |`dependency-deprecation-helper`| Ready | Located under `fixtures/modules/synthetic-deprecation-helper`. |
108
108
109
109
### Coverage matrix
110
110
@@ -131,23 +131,23 @@ The comparison harness lives under `scripts/check-modules/compare/` and is exerc
131
131
132
132
### Goals
133
133
134
-
- Execute legacy (`--checks=legacy`) and TypeScript implementations against the same curated dataset.
135
-
- Normalize and diff Stage 5 outputs, surfacing rule-level differences and aggregate stats.
134
+
- Execute one or two configured commands against the curated dataset.
135
+
- Normalize and diff Stage 5 outputs, surfacing rule-level differences and aggregate stats when two runs are available.
136
136
- Emit machine-readable (`diff.json`) and Markdown (`diff.md`) reports for CI artifacts.
137
137
138
138
### Flow
139
139
140
140
1. Prepare a temporary workspace with the curated fixtures.
141
-
2. Run the legacy stage via the orchestrator, capturing Stage 5 outputs.
142
-
3. Run the TypeScript stage (default) in a separate directory.
141
+
2. Run the first configured command (labelled `legacy` by default) and capture Stage 5 outputs.
142
+
3. Run the second command (`ts` by default) if provided.
0 commit comments