Evaluation engineer. I audit frontier coding benchmarks for construct validity, finding where the headline metric measures something other than what it claims. Every audit ships as a preregistered, re-runnable artifact with a public repo. This page links the receipts.
| benchmark | finding | receipts |
|---|---|---|
| SWE-bench Pro | 15% of the 728 public tasks are underdetermined: the hidden tests grade behavior the spec never stated. 3 gold patches fail the benchmark's own verifier. | audit · tool |
| SWE-bench Pro (run) | 95.3% (694/728) under the official grader, solo, mostly on a $200/month plan. Preregistered, frozen, every verdict re-gradable from a committed diff. | repo · field guide |
| ProgramBench | "% Resolved" scores recall of published algorithms, not source-blind reconstruction. 21+ programs are gated on recalling a hash, cipher, or codec. | paper · repo |
| Terminal-Bench | Grading is blind to destruction. A run can wreck everything outside the task frame and still pass. | paper · repo |
| DeepSWE | 4 of 113 reference solutions fail their own verifiers. Found for under $1, in under an hour, with a preregistered two-pass protocol. | v1 · v1.1 · repo |
| SWE-bench Verified | 426/500 (85.2%), denominator reconciled instance by instance; the set is contamination-compromised. | post · repo |
| SWE-rebench | Determinacy audit of the 2026_03 split: a 14.5% pointer-checkable claimable spine. | repo |
- The Hypothesis Graph, Verifiable Knowledge, and What Cannot Be False Cannot Be True: DOI-archived preprints with reproducible artifacts.
- Speedrunning Open Source: adversarial review loops push test-passing LLM code from 43% to 91% merge-readiness, deployed as real-maintainer PRs.
- Methodology in public: a preregistration checklist, published null results, and a post-mortem of a $1,000 mistake caused by held-out-test leakage.
112 merged PRs across 90 repos I don't own, in Rust, Go, C++, and Python: godot, hyper, envoy, servo, tidb, Enzyme, flux, wild, and 82 more. It's how I read a benchmark's test suite in whatever stack it ships.
Since the pipeline epoch (2026-05-09): 88 merged / 182 resolved, a 48% merge rate, with 104 still open as of Jul 9, 2026.
the funnel, triage to merge · as of Jul 9, 2026
sankey-beta
triaged, submitted, 322
triaged, throttled, 36
triaged, rejected, 10
submitted, resolved, 182
submitted, dripped, 36
submitted, open, 104
resolved, merged, 88
resolved, closed, 94
Triage-side counts are frozen at campaign end (May 20); the submitted branch is live GitHub data, with dripped as the remainder.
where the closed PRs went
Most closed PRs are self-withdrawals, no-AI policies, duplicates, or bot closes; the audit is in why the closed PRs closed, with machine-readable receipts in pr-receipts.jsonl and closed-pr-reasons.jsonl. A sibling campaign filed 65 issues offering repos a slop filter; 67% positive reception among maintainer-decided, tracked in a hypothesis graph (as of May 20).
recent feed · ✅ merged, ❌ closed unmerged
| repo | PR | |
|---|---|---|
| ❌ | FyroxEngine/Fyrox | #917 Fix transmute_slice UB: add alignment, divisibility, and ZST guards |
| ✅ | akshettrj/watgbridge | #85 fix: apply ephemeral timer to @all and .id commands |
| ❌ | pola-rs/polars | #27561 fix(rust): Return correct Struct dtype from qcut on empty series |
| ❌ | feldera/feldera | #6219 feat(pytest): add read_table() and log_files() to DeltaTestLocation |
| ❌ | open-webui/open-webui | #24547 fix: llamacpp load/unload indicator now detects loaded models via /slots |
| ❌ | SWE-bench/experiments | #448 Add Verified submission: recon-craft-audit (426/500, 85.2%) |
| ✅ | EnzymeAD/Enzyme | #2899 [MLIR] fix: complex.create reverse mode flips imaginary-operand gradient sign |
| ❌ | oven-sh/bun | #30430 Fix per-stream ANSI color detection when stdout/stderr differ |
hall of ❌: PRs closed for being AI, with zero bugs found
| PR | time to close | bugs | title |
|---|---|---|---|
| uptime-kuma#7371 | <1 min | 0 | 🚨 |
| uptime-kuma#7372 | <1 min | 0 | 🚨 |
| litestar#4755 | 7 hrs | 0 | closed per AI policy |
| ruff#25066 | 2 days | 0 | mainly produced by AI |
| llama.cpp#22873 | 2 days | 1 | AI-generated PR detected |
leaderboard · voluntary contributions to repos you don't own · as of Jul 9, 2026
| contributor | merged | rate | repos | median diff |
|---|---|---|---|---|
| mvanhorn | 715 | 66% | 404 | 80 |
| SAY-5 | 489 | 73% | 447 | 23 |
| fdelbrayelle | 329 | 91% | 169 | 1 |
| ununununium | 84 | 72% | 64 | 1 |
| kimjune01 | 70 | 50% | 63 | 45 |
PRs created since 2026-05-09 to repos outside the author's account; rate = merged ÷ (merged + closed unmerged), median diff = added + deleted lines. The kimjune01 row is lower than the funnel above because the funnel includes own-account repos. yakushabb, on the May board with 24 merged, dropped off: the account no longer exists.
verify the numbers yourself
{ merged: search(query: "is:pr is:merged author:kimjune01 created:>2026-05-09T00:34:00Z", type: ISSUE) { issueCount }
closed: search(query: "is:pr is:closed is:unmerged author:kimjune01 created:>2026-05-09T00:34:00Z", type: ISSUE) { issueCount }
allTime: search(query: "is:pr is:merged author:kimjune01 -user:kimjune01", type: ISSUE) { issueCount } }