Skip to content

Commit 6a519a2

Browse files
hyperpolymathhyperpolymath
authored
docs(audits): workflow convergence campaign 2026-05-26 (#199)
## Summary Estate-wide audit doc for the workflow convergence campaign just completed (5 reusable-extraction PRs, methodology gotcha discovery, parallel-session reconciliation). - 4 reusables already filed: #187 (mirror), #190 (secret-scanner), #192 (codeql), #193 (hypatia-scan) - Classifier tooling shipped: #194 - Scorecard-reusable recommended as #195 (cleanest sweep target — zero nested copies, 38.8% top-SHA dominance, 100% mechanical drift) ## Key findings documented 1. **Nested-path methodology gotcha** (3-layer): - `path:.github/workflows` filter excludes nested workflows (mirror +133, secret-scanner +282, codeql +518 verified per-repo, hypatia-scan +449) - Even broad `filename:` queries are org-scope-truncated; per-repo recursive-tree queries are the only reliable source of truth - **Nested workflows do NOT execute** — GitHub Actions only runs `.github/workflows/` at the repo root. Nested copies are inert vendored templates or stale leftover, not active deployments 2. **Corrected estate counts** (top-level + nested true totals): - hypatia-scan: 704 wrapper sites (×416 LOC ≈ 280k retired) - codeql: ~781 wrapper sites (×41 LOC ≈ 32k retired — 3× original estimate) - secret-scanner: 563 wrapper sites - mirror: 422 wrapper sites - scorecard: 258 (no nesting) 3. **Parallel-session ranking divergence** root-caused — sample-drift methodology missed homogeneity findings for secret-scanner / codeql / scorecard. ## Test plan - [ ] Validate cross-refs render (`/.claude/CLAUDE.md`, sibling audit docs) - [ ] No breakage — additive doc only ## Standing follow-ups (documented in body) 1. Owner-review the 5 open campaign PRs 2. File #195 (scorecard-reusable) to close the set 3. Extend classifiers in #194 to emit per-(repo, path) tuples for nested-copy sweeps 4. Wrapper sweep firing remains owner-gated per template 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent d609387 commit 6a519a2

1 file changed

Lines changed: 213 additions & 0 deletions

File tree

‎docs/audits/workflow-convergence-campaign-2026-05-26.md‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
# Workflow Convergence Campaign — 2026-05-26
2+
3+
> Generated by: estate-wide audit of drifting per-repo workflow templates
4+
> Scope: 5 candidate templates ranked by drift × deployments × feature-variance
5+
> Outcome: 5 reusable-workflow PRs filed in this repo; classifier tooling shipped; nested-path methodology gotcha documented for future campaigns
6+
7+
## Summary
8+
9+
This campaign extracted 5 drifting per-repo workflow templates into reusable workflows hosted in this repo, plus shipped the classifier tooling and a Git-Tree-walk helper that planned the wrapper sweeps and validated the deployment counts.
10+
11+
The campaign also surfaced a **methodology gotcha** that affects every future drift survey in this estate: `gh api /search/code` queries undercount workflow files in three compounding ways. After the campaign-meta-doc filed its first cut, the helper at [#204](https://github.com/hyperpolymath/standards/pull/204) walked the Git Tree API for all 5 templates and revised the count tables — top-level deployments were undercounted by 1–35% depending on template, and nested copies were undercounted by 100%+ for several templates. See [Corrected estate counts](#corrected-estate-counts) for the helper-validated tables.
12+
13+
## Filed PRs
14+
15+
Deploy counts here are the **helper-validated top-level counts** (from `list-workflow-paths.sh` walking each repo's Git Tree). The initial path-filtered survey numbers — which are listed in the PR bodies themselves — are 1–35% lower depending on template.
16+
17+
| PR | Template | Top-level (helper) | Top-level (PR body) | Reusable LOC | Top SHA share |
18+
|---|---|---:|---:|---:|---|
19+
| [#187](https://github.com/hyperpolymath/standards/pull/187) | `mirror-reusable.yml` | 293 | 289 | 165 | 76% |
20+
| [#190](https://github.com/hyperpolymath/standards/pull/190) | `secret-scanner-reusable.yml` | 299 | 281 | 159 | 69% (across top 4 SHAs) |
21+
| [#192](https://github.com/hyperpolymath/standards/pull/192) | `codeql-reusable.yml` | 280 | 263 | 96 | 83% single-language |
22+
| [#193](https://github.com/hyperpolymath/standards/pull/193) | `hypatia-scan-reusable.yml` | 344 | 255 | 459 | 83.5% top-5 |
23+
| [#194](https://github.com/hyperpolymath/standards/pull/194) | sweep-classifier scripts | — | — | — | tooling |
24+
| [#199](https://github.com/hyperpolymath/standards/pull/199) | this campaign meta-doc | — | — | — | docs |
25+
| [#204](https://github.com/hyperpolymath/standards/pull/204) | `list-workflow-paths.sh` helper + classifier ingestion | — | — | — | tooling |
26+
| [#205](https://github.com/hyperpolymath/standards/pull/205) | `scorecard-reusable.yml` | 278 | 258 | 87 | 38.8% |
27+
28+
All PRs have auto-merge enabled (per the [auto-merge-on-all-PRs standing policy](https://github.com/hyperpolymath/.github)). The wrapper sweep does NOT fire automatically — each template is owner-gated post-merge.
29+
30+
## Convergence set status
31+
32+
The 5-candidate convergence set is **fully filed** (#187 mirror / #190 secret-scanner / #192 codeql / #193 hypatia-scan / #205 scorecard). #194 (classifiers), #199 (this doc), and #204 (helper + nested-path classifier ingestion) close out the supporting infrastructure.
33+
34+
## Ranking methodology
35+
36+
For each candidate, three signals were combined:
37+
38+
1. **Drift** = (unique blob SHAs / total deployments). Lower = more homogeneous.
39+
2. **Deployments** = number of repos carrying the template. Higher = more leverage.
40+
3. **Feature variance** = whether the SHAs differ in job-set / step-set / language matrix (real customization) or only in SPDX-header / action-pin / whitespace (mechanical lag).
41+
42+
A reusable is viable when (2) is high AND (3) is low. (1) tells you how much wrapper-sweep work the rollout needs.
43+
44+
| Template | Deploys | Drift | Feature variance | Reusable viable |
45+
|---|---|---|---|---|
46+
| `hypatia-scan.yml` | 255 | 11.8% | ZERO | ✅ (filed #193) |
47+
| `scorecard.yml` | 258 | 17.8% | ZERO | ✅ (recommended #195) |
48+
| `secret-scanner.yml` | 281 | 19% | LOW (job-set homogeneous; force-propagates `shell-secrets` guardrail) | ✅ (filed #190) |
49+
| `mirror.yml` | 289 | 24% | LOW (job-set homogeneous; same 7 forges across top SHAs) | ✅ (filed #187) |
50+
| `codeql.yml` | 263 | 26% | MEDIUM (language matrix variance — needed `language` + `build-mode` inputs) | ✅ (filed #192) |
51+
52+
## The nested-path methodology gotcha
53+
54+
### Layer 1: path-prefix filter excludes nested workflows
55+
56+
`gh api /search/code` with `path:.github/workflows` matches the path **PREFIX**, so:
57+
- `.github/workflows/codeql.yml` ✓ matches
58+
- `developer-ecosystem/asdf-augmenters/.github/workflows/codeql.yml` ✗ does NOT match
59+
60+
Removing the `path:` filter and running `filename:codeql.yml org:hyperpolymath` exposes nested copies inside monorepos like `developer-ecosystem`, `ssg-collection`, `ambientops`, `standards`, `julia-ecosystem`, `asdf-tool-plugins`, etc.
61+
62+
### Layer 2: even the broad query is org-scope-truncated
63+
64+
GitHub Code Search's org-scoped result set is capped well below the true file count. For `codeql.yml`:
65+
66+
| Monorepo | Broad query saw | Per-repo query (truth) | Missed |
67+
|---|---:|---:|---:|
68+
| developer-ecosystem | 41 | 170 | 129 |
69+
| asdf-tool-plugins | 11 | 111 | 100 |
70+
| ssg-collection | 39 | 68 | 29 |
71+
| standards | 26 | 39 | 13 |
72+
| ambientops | 12 | 29 | 17 |
73+
| julia-ecosystem | 11 | 20 | 9 |
74+
| **Top-6 total** | **140** | **437** | **297** |
75+
76+
Broad-query undercount factor: **~2.6×** inside heavily-nested monorepos. Per-repo queries (`gh api repos/<org>/<repo>/git/trees/HEAD?recursive=1 --jq '.tree[] | select(.path | endswith("/<file>.yml"))'`) are the only reliable source of truth for monorepo-nested workflow files.
77+
78+
**Layer 2 also affects path-filtered queries** (a finding from the helper re-survey). `path:.github/workflows filename:hypatia-scan.yml` returned 255 results, but the Git-Tree walk found 344 top-level files — a 35% gap on the most-deployed template. The path filter is less severely truncated than the broad query, but both are sub-truth. Only direct Git-Tree enumeration is reliable.
79+
80+
### Layer 3: nested workflows are inert
81+
82+
**GitHub Actions only runs workflows from the repo-root `.github/workflows/` directory.** Workflows under any other path (e.g. `a2ml/bindings/deno/.github/workflows/secret-scanner.yml` inside the `standards` repo) are **never triggered**.
83+
84+
This means nested copies are one of:
85+
86+
1. **Vendored templates** — maintained as a snapshot of what a sub-package's own workflow would be if it were extracted into its own GitHub repo. Read-only intent; not load-bearing for the sub-package's CI.
87+
2. **Stale leftover** — surviving artifacts of a previous monorepo merger; pure dead code.
88+
89+
**Implication:** the security-gap argument that motivated the secret-scanner reusable (forcing `shell-secrets` propagation post-Cloudflare-leak) does NOT extend to nested copies. The ~282 nested `secret-scanner.yml` files do not represent missing-guardrail attack surface — they don't run. The wrapper sweep on nested copies is **single-source-of-truth cleanup**, not security-hardening.
90+
91+
### Corrected estate counts
92+
93+
After [#204](https://github.com/hyperpolymath/standards/pull/204) shipped `list-workflow-paths.sh`, all 5 templates were re-enumerated by walking each repo's Git Tree API directly. The results invalidated **both** the broad-query and path-filtered survey numbers — even top-level counts were undercounted across the board.
94+
95+
#### Helper-validated counts (Git-Tree walk, all 5 templates)
96+
97+
| Template | Top-level | Nested | Total | Unique blob SHAs (all) |
98+
|---|---:|---:|---:|---:|
99+
| `hypatia-scan.yml` | **344** | 603 | **947** | 32 |
100+
| `mirror.yml` | **293** | 335 | **628** | 120 |
101+
| `secret-scanner.yml` | **299** | 292 | **591** | 83 |
102+
| `codeql.yml` | **280** | 646 | **926** | 175 |
103+
| `scorecard.yml` | **278** | 626 | **904** | 114 |
104+
105+
#### Top-level-only drift
106+
107+
The drift % when nested copies are excluded is the figure relevant for the *executing* surface (since nested workflows are inert, per Layer 3):
108+
109+
| Template | Top-level | Unique blob SHAs (top-level only) | Drift |
110+
|---|---:|---:|---:|
111+
| `hypatia-scan.yml` | 344 | **3** | **0.9%** |
112+
| `secret-scanner.yml` | 299 | 54 | 18.1% |
113+
| `scorecard.yml` | 278 | 46 | 16.5% |
114+
| `mirror.yml` | 293 | 75 | 25.6% |
115+
| `codeql.yml` | 280 | 75 | 26.8% |
116+
117+
**hypatia-scan top-level is byte-near-identical across 344 sites** — only 3 unique blob SHAs total. The reusable wrapper sweep for hypatia-scan is therefore essentially mechanical with near-zero per-repo variance, much tighter than the 11.8% drift the PR body reports (which was the drift across top-level + nested).
118+
119+
#### Initial-survey undercount summary
120+
121+
| Template | PR-body top-level | Helper top-level | Top-level undercount | Original nested estimate | Helper nested |
122+
|---|---:|---:|---:|---:|---:|
123+
| `hypatia-scan.yml` | 255 | 344 | **+89 (35%)** | 449 | 603 |
124+
| `secret-scanner.yml` | 281 | 299 | +18 (6%) | 282 | 292 |
125+
| `codeql.yml` | 263 | 280 | +17 (6%) | ~518 | 646 |
126+
| `scorecard.yml` | 258 | 278 | +20 (8%) | 626 | 626 |
127+
| `mirror.yml` | 289 | 293 | +4 (1%) | 133 | 335 |
128+
129+
The `path:.github/workflows`-filtered query under-reports top-level counts at the rate above, and the broad query under-reports nested counts even more severely (most extremely for mirror: 133 reported vs 335 true). Some of the gap is creation-of-new-repos between survey-and-helper runs, but most is search-index truncation.
130+
131+
#### LOC retirement (top-level only — the executing surface)
132+
133+
| Template | Top-level | × Canonical LOC | LOC retired |
134+
|---|---:|---:|---:|
135+
| `hypatia-scan.yml` | 344 | 416 | **~143,000** |
136+
| `mirror.yml` | 293 | 145 | ~42,500 |
137+
| `codeql.yml` | 280 | ~150 | ~42,000 |
138+
| `secret-scanner.yml` | 299 | ~120 | ~36,000 |
139+
| `scorecard.yml` | 278 | 41 | ~11,400 |
140+
| **Total (5 reusables, top-level)** | | | **~275,000** |
141+
142+
Including nested copies (single-source-of-truth cleanup, not the executing surface): **~732,000 LOC** estate-wide eligible for wrapper replacement.
143+
144+
## Classifier tooling pattern (`scripts/sweep-classifiers/`)
145+
146+
Each template has a `classify-<template>.sh` script (committed in [#194](https://github.com/hyperpolymath/standards/pull/194)) that:
147+
148+
1. Reads paginated `gh api /search/code` JSON into a `(repo, sha)` table.
149+
2. Fetches each unique blob ONCE (cached in `$BLOBS_DIR`) — bandwidth-efficient.
150+
3. Classifies each blob by job-set / line-count / language-matrix / SPDX-header lag.
151+
4. Emits a per-repo TSV with `<repo> <class> <evidence>` for sweep batching.
152+
153+
Pattern: the classifier output drives a 3-wave sweep (mechanical bulk → propagation-lag bulk → per-repo review). Wave-1 PRs are auto-generated wrappers pinned to the merged-reusable HEAD SHA.
154+
155+
**Outstanding tooling gap** (filed as follow-up): classifiers currently emit one row per repo, not per `(repo, path)` tuple. For monorepo-nested copies, the classifiers need to ingest the per-repo recursive-tree listing and emit one wrapper site per nested copy. This is a small change to each script's input loop.
156+
157+
## Per-template specifics
158+
159+
### `mirror.yml` (PR #187)
160+
161+
7 forge jobs (gitlab, bitbucket, codeberg, sourcehut, disroot, gitea, radicle), each gated on `vars.<FORGE>_MIRROR_ENABLED == 'true'`. Only input: `runs-on`. Caller MUST use `secrets: inherit` (jobs reference forge-specific secrets). Wrapper sweep is mostly mechanical; the 22 NEEDS_REVIEW cases in the classifier output are slim 2-3-forge variants (older repos pre-7-forge expansion) or NONE-stub placeholders.
162+
163+
### `secret-scanner.yml` (PR #190)
164+
165+
4 jobs: `trufflehog`, `gitleaks`, `rust-secrets`, `shell-secrets`. Self-conditioning — `rust-secrets` exits early on no `Cargo.toml`, `shell-secrets` no-ops without `.sh`/`.bash`. **Force-propagates** the `shell-secrets` job (post-Cloudflare-leak guardrail) to the estate, where it currently exists in only 1 of 281 top-level deployments (this repo itself). The 282 nested copies do NOT close additional security gaps (Layer 3 caveat) but DO consolidate the source of truth.
166+
167+
The full-pagination data corrected an initial 100-repo sample's "55% drift" reading to a true **19% drift** with 100% job-set homogeneity. The parallel-session ranking that said "skip secret-scanner" was based on the sample drift figure and missed the homogeneity finding.
168+
169+
### `codeql.yml` (PR #192)
170+
171+
Two inputs: `language` (default `javascript-typescript`) and `build-mode` (default `none`). Single-language single-job design — multi-language wrappers invoke the reusable per-language. Language matrix distribution across 263 top-level deployments:
172+
173+
- 84.8% `javascript-typescript`-only
174+
- 8.4% `actions`-only
175+
- ~3.4% multi-language
176+
- ~3.4% NONE-matrix (needs per-repo review)
177+
178+
The campaign's most complex reusable — but still ~93% mechanical sweep.
179+
180+
### `hypatia-scan.yml` (PR #193)
181+
182+
Byte-identical to the canonical 416-line `scan` job. Single `runs-on` input. Caller MUST grant `security-events: write` + `pull-requests: write` permissions AND use `secrets: inherit`. The 416-line canonical × 704 total estate sites = ~280k LOC retired (the largest leverage of the 5 campaigns). 30 unique blob SHAs / 11.8% drift / 100% single-`scan`-job homogeneity. All drift is pure propagation lag (older slimmer versions of the same body).
183+
184+
### `scorecard.yml` (recommended #195)
185+
186+
258 deployments, 17.8% drift, **100% mechanical drift** — every variant has one `analysis` job with `checkout → ossf/scorecard-action → upload-sarif`. Same single `scorecard-action` SHA pin across all variants; differences are SPDX header (`PMPL-1.0` / `PMPL-1.0-or-later` / `MPL-2.0`), `upload-sarif` SHA pin, `permissions: read-all` vs `contents: read`, and presence/absence of the estate concurrency-guardrail block. 41-line canonical → ~8k LOC retired top-level only. Zero nested copies, making it the cleanest sweep candidate.
187+
188+
## Reconciliation with the parallel-session ranking
189+
190+
A parallel session ([[project_foundational_workflow_survey_2026_05_26]] in agent memory) ranked the 5 candidates and concluded "pursue hypatia-scan + mirror; skip codeql / secret-scanner / scorecard." The recommendation set differs from this campaign's outcome on three templates:
191+
192+
| Template | Parallel call | This campaign | Reason for divergence |
193+
|---|---|---|---|
194+
| `secret-scanner.yml` | SKIP | PURSUE | Parallel based on 55% sample drift; full pagination → 19% drift, 100% homogeneity. The skip call missed the shell-secrets propagation gap. |
195+
| `codeql.yml` | SKIP | PURSUE | Real (but contained) language-matrix variance — 85% are single-language `javascript-typescript`, making the 2-input reusable dead-simple for the dominant case. |
196+
| `scorecard.yml` | SKIP | PURSUE | Same shape as secret-scanner mistake: 18% sample drift looks like real customization but is ALL mechanical (action-SHA churn + SPDX header drift). |
197+
198+
The agreement between sessions was strong on `hypatia-scan.yml` (both: pursue, HIGH homogeneity) and `mirror.yml` (both: pursue, ~76% top-SHA dominance). The divergence comes from the parallel session's reliance on sample-drift estimates without full pagination, where my campaign full-paginated and blob-fetched each candidate.
199+
200+
## Standing follow-ups
201+
202+
1. **Owner-review on the 8 PRs** (#187/#190/#192/#193/#194/#199/#204/#205) — auto-merge enabled; CI-green will auto-land.
203+
2. **Per-repo recursive-tree classifier inputs** — DONE in #204. Classifiers now accept the helper's TSV output (`<repo>\t<path>\t<blob-sha>`) and emit one row per `(repo, path)` tuple. Backward-compatible with legacy JSONL.
204+
3. **Wrapper sweep firing** is owner-gated per template. The 82-PR rust-ci sweep is the upper bound the user has signalled comfort with; mirror's 293-PR sweep would need explicit per-batch sign-off. With the helper-validated counts, sweeps now have ground-truth target lists.
205+
4. **Nested-copy disposition decision** — for each monorepo (developer-ecosystem, ssg-collection, ambientops, julia-ecosystem, asdf-tool-plugins, etc.), are the nested workflows vendored templates (keep, sync to reusable) or stale leftover (delete)? This is a per-monorepo judgement call; the campaign defers it.
206+
5. **Re-run helper periodically** — the Git-Tree-walk truth shifts as new repos land. Suggest a quarterly re-run of `list-workflow-paths.sh` against each of the 5 templates to keep this doc's count tables fresh.
207+
208+
## Cross-references
209+
210+
- Prior pattern: PR [#168 language-policy convergence](https://github.com/hyperpolymath/standards/pull/168) — first instance of "subsume legacy workflow into governance-reusable" pattern; merged 2026-05-26.
211+
- Prior pattern: PR [#174 rust-ci + elixir-ci reusables](https://github.com/hyperpolymath/standards/pull/174) — extended the pattern; 82-PR wrapper sweep filed downstream (`project_rust_ci_wrapper_sweep_2026_05_26`).
212+
- Estate-wide language policy: [`.claude/CLAUDE.md`](../../.claude/CLAUDE.md) — Python/V-lang/ATS2/ReScript/TypeScript bans enforced by `governance-reusable.yml`.
213+
- Security context: the `shell-secrets` job force-propagation in #190 is direct mitigation for the [unrotated Cloudflare API token leak](#) (memory `project_cloudflare_token_leak_2026_05_21`).

0 commit comments

Comments
 (0)