Skip to content

Running the gates locally does not cover everything CI runs: the gate-script tests are in no gate and no hook, and they are red right now #1488

Description

@DmitriyG228

Value this issue delivers

A contributor's local green predicts CI's green. Running the gates locally covers every static
check CI will run — or CI's extra step is named, with its reason, so nobody is surprised by a red
they had no way to see.

Why this matters

CI's static job runs the thirteen pnpm gate:* steps and then one more:

- name: gate script tests
  run: node --test scripts/*.test.mjs release/*.test.mjs

These are the tests of the gate scripts — the instruments every other step trusts. The step's
own comment says why it is there: "scripts/ is not a workspace package, so pnpm test — turbo run
test — never reaches it: this step is the only lane these files run in."

It is not one of the gates in scripts/gates.mjs's GATES table, and it is not in the pre-push
hook's FAST_GATES list. So there is no local command that runs it as part of "the checks":
pnpm gates does not, git push does not, and pnpm test explicitly does not. The only way a
contributor runs it is by typing that node --test line, which they can only know about by reading
the workflow file.

That gap is not theoretical, and it is worse than "you find out in CI". At the head of this
branch's tree, the step is RED
, deterministically, and no local check in the repository reports
it (see below). A lane nobody runs is a lane that rots, and this one guards the instruments — a
broken gate-script test is precisely the thing that lets a broken gate through.

The hook's own stated bar is the argument for closing this: it exists "to catch cheap structural
mistakes before they burn a CI round-trip"
. This step is cheap and structural. If it turns out not
to be cheap enough, that is a decision worth writing down rather than leaving implicit.

Where we are (honest)

fact where
the step, its command and its rationale comment .github/workflows/gates.yml:51-52
the GATES table — no entry for it, so node scripts/gates.mjs <name> cannot run it scripts/gates.mjs, the GATES const
FAST_GATES — fourteen names, none of them this .githooks/pre-push
what the step covers: 11 files, 133 tests scripts/*.test.mjs (8) + release/*.test.mjs (3)
measured wall time on the shared build host, warm: ~3.9 s (real 0m3.965s) time node --test scripts/*.test.mjs release/*.test.mjs
it is currently RED, 132 pass / 1 fail, reproducibly (three consecutive runs, same failure) scripts/sbom.test.mjs:53"a newly declared Lite final-stage apt package is automatically inventoried", failing its own fixture assertion fixture setup did not alter Dockerfile.lite
the same test passes when its file is run alone (node --test scripts/sbom.test.mjs → 2/2)
root cause of that red: gates.test.mjs's withEdited() mutates tracked files in the working tree (deploy/lite/Dockerfile.lite, deploy/compose/docker-compose.yml, values.yaml, image-licenses.json) and restores them in a finally; node --test runs the files as parallel processes, so sbom.test.mjs reads Dockerfile.lite while gates.test.mjs has it edited scripts/gates.test.mjs:130-138, scripts/sbom.test.mjs:53

Two honest notes on that red. First, it is not caused by this repository's flows/compose work
deploy/lite/Dockerfile.lite, scripts/sbom.mjs and scripts/sbom.test.mjs are byte-identical to
a5ba8e952. Second, if it reproduces this reliably here, the first thing to check is whether
CI's static job is currently red on this step
— a different node version or scheduler could
mask it there, which would be its own finding: the step is green in the one place it runs and red
everywhere else, or red in CI and nobody noticed because it is the last step of a job people read
as "the gates".

The ~3.9 s measurement matters for the decision, not just as trivia: as it stands the step does not
clear a 2-second pre-push bar, so the choice is real rather than rhetorical.

Deployments to validate (D12b)

  • Not a deployment change. This is contributor tooling and CI wiring; the observation is what a
    local command prints and what CI runs.
  • The attestation names the machine and its node version, because the step's cost and its parallel
    scheduling are both properties of that.

Docs surface (D6c)

  • .githooks/pre-push — reference altitude, in the file's own comment block: whatever the answer
    is, the FAST_GATES list should be readable as complete with respect to CI's static job, or
    should say what it deliberately omits and why.
  • scripts/gates.mjs — reference altitude: if the step becomes a gate, its usage line and the
    GATES table are the place it becomes discoverable.
  • CONTRIBUTING.md (or wherever "run the checks before you push" is written) — how-to altitude:
    one command a contributor can trust.
  • Argued as no-change: docs/docs/governance/delivery.mdx — this changes which lane runs a check,
    not the delivery standard.

The components (validation waypoints of the ONE PR)

  • C1 — the gate-script tests are either a gate that runs pre-push, or documented CI-only

C1 · the gate-script tests are either a gate that runs pre-push, or documented CI-only

Target: ONE seam — the gate runner's coverage surface: scripts/gates.mjs (the GATES table)
and .githooks/pre-push (FAST_GATES).

Value: one local command covers CI's static job, or the delta is named.
Prepared solution: add a gate:script-tests entry that runs
node --test scripts/*.test.mjs release/*.test.mjs, put it in FAST_GATES, and have the workflow
call the gate rather than the raw command so the two cannot drift. If the measured cost keeps it
out of the hook
, take the other branch deliberately: leave it CI-only, and say so in
.githooks/pre-push's comment beside the FAST_GATES list, naming the cost and the command a
contributor can run by hand. Either outcome closes this issue; a silent status quo does not.
Along the way:

  • The lane is red today (sbom.test.mjs, root-caused above). Making it a gate makes that red
    block pushes, so it has to be fixed first or the change lands broken. Fixing it is arguably a
    separate issue — gates.test.mjs mutating tracked files in the working tree while node --test
    runs files in parallel is a test-isolation defect with two consequences beyond this flake: a run
    killed mid-test leaves the developer's tree modified, and two concurrent runs (or a concurrent
    agent on the same checkout) corrupt each other. Decide split-or-bundle explicitly; do not
    discover it half-way.
  • Do not make it a gate that can pass vacuously. If the glob matches nothing the command
    succeeds; the gate should count the files it ran, the way the others report their counts.
  • Watch the double-run cost: the hook runs the fast set, and CI runs it again. That is the
    existing bargain for the other fourteen and is not a reason to skip this one.
  • -x / fail-fast semantics differ between node --test and the pytest-based legs; keep the
    failure output as legible as the current step's.
    Early validation: time node scripts/gates.mjs script-tests at the new gate's name, and a
    git push on a throwaway branch showing the step in the hook's output.

The acceptance table — present these observations and your PR merges

# Observation Negative control (shown RED) Anchor
A1 The step is covered or the gap is named. EITHER: node scripts/gates.mjs script-tests exists and reports the file/test counts, the name appears in .githooks/pre-push's FAST_GATES, a real git push shows it running, and it completes in under 2 s — OR: .githooks/pre-push carries an explicit comment listing this step as CI-only with its measured cost as the reason, and CI calls the gate name rather than the raw command. A deliberately broken gate-script test: the push is refused (gate branch), or the documented CI-only note is shown to be the only thing standing between that break and a green local run (documented branch). head sha · time output · push transcript or the comment diff
A- No-regression: the other fourteen fast gates still run and still pass at head; CI's static job proves the same set it proves today. CI at head sha

How this issue closes (the live validation — the human part)

One contributor, no second person: break one assertion in scripts/gates.test.mjs, run whatever
the repo now says is "the checks before you push", and see it caught — or see the note that tells
you it will only be caught in CI, and why. The docs story is part of the signature: after this
change there should be exactly one command a newcomer is told to run, and it should be true.

Authorship

Found during #1471's run — a local green did not predict CI's static job. Prepared with an agent;
no code in this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state: preparedFull standard shape present; awaiting the maintainer ready-stamp

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions