Add the writeup

Lives in docs/, linked from the README by one line rather than expanded into a
section. The README's job is to make someone want to use the tool; this piece's
job is to explain why it exists. Mixing them is how a real artifact starts
reading as a demo.

Leads with the thing anyone can check for themselves: a 50-case suite going 82%
to 78% produces a 95% interval spanning [-0.16, +0.00] at p = 0.63, which is no
evidence in either direction, and most teams would have reverted on it.

The core of it is section 2, the cluster undercoverage, because it is the one
claim here that could not have been produced by argument. The pairs cluster
bootstrap is the obvious method, I built it, and calibration measured it at 89.5%
coverage for a nominal 95%. It needs ~50 clusters to be honest and real eval
suites have eight. That is a finding, not a design opinion, and the measured
table is in the piece.

Every quoted output is verbatim from the real code, re-run and diffed rather than
transcribed from memory. Every number cross-checks against inference.py.

The AI-assistance note is deliberately in this file and not in the README, is
specific enough to check against the commit trailers and the diff, and does not
claim more for me than is true: the code was largely generated, the scope
decision was mine, and the bug in section 2 was caught by a simulation rather
than by anyone reading anything.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: Skits0098

README.md

@@ -193,10 +193,17 @@ statisticians.
   20-50 real-failure cases as a starting suite; regression evals sit near 100%
   and exist to detect backsliding.
 
+## Reading
+
+- [**Most of your eval regressions aren't real**](docs/most-eval-regressions-arent-real.md)
+  - why a 50-case suite cannot answer the question you are asking it, and what the
+  simulations caught that code review would not have.
+- [DESIGN.md](DESIGN.md) - the full design, and decision records for the two
+  choices I got wrong first.
+
 ## Status
 
-Pre-release, v0.1 in progress. See [DESIGN.md](DESIGN.md) for the full design and
-the reasoning behind each decision, including the ones I got wrong first.
+Pre-release, v0.1 in progress.
 
 ## License
 

docs/most-eval-regressions-arent-real.md

@@ -0,0 +1,328 @@
+# Most of your eval regressions aren't real
+
+You change a prompt. Your eval suite goes from 82% to 78%.
+
+What do you do?
+
+Most people revert. Some ship anyway and feel bad about it. Almost nobody asks the
+only question that matters, which is whether 78% and 82% are different at all.
+
+Here is that exact comparison, run through the tool I will describe below:
+
+```
+verdict:   INCONCLUSIVE
+effect:    -0.0400 (0.8200 -> 0.7800)
+95% CI:    [-0.1600, +0.0000]
+p:         0.625  (exact McNemar)
+paired:    50 cases
+```
+
+Four of your fifty cases moved. The interval spans everything from a sixteen point
+regression to no change whatsoever. McNemar puts it at p = 0.63. There is no
+evidence here in either direction. You learned nothing, and if you reverted, you
+reverted on a coin flip.
+
+This is not a contrived example. It is what a 50-case suite looks like, and 50
+cases is what Anthropic's own guidance suggests you start with.
+
+## Nothing in your toolchain can tell you this
+
+I went looking for a tool that would. promptfoo, DeepEval, Ragas, Braintrust
+autoevals, Langfuse, Phoenix, pydantic-evals. Every one of them computes a mean
+and compares it to a threshold you hardcoded.
+
+`assert score > 0.8`
+
+That comparison is close to meaningless on a suite this size, and it is the actual
+mechanism behind the thing everyone complains about: flaky eval CI. Your suite
+reports 82% today and 78% tomorrow, the tool says REGRESSION with total
+confidence, and you go hunting for a bug that was never there. Then it happens
+often enough that you stop believing the tool, which is worse than not having it.
+
+The field has good data collection. OpenTelemetry tracing is solved, and every
+observability vendor will happily store your spans. It has good scoring
+primitives: judges, assertions, rubrics. What it does not have is any statistics
+between "we ran an eval" and "we know whether this change is real."
+
+## The fix is twenty months old and nobody implemented it
+
+Evan Miller, an econometrician at Anthropic, published
+[*Adding Error Bars to Evals*](https://arxiv.org/abs/2411.00640) in November 2024.
+It is not a research breakthrough. It is a patient explanation that eval scores
+are a *sample*, that samples have standard errors, and that undergraduate
+experimental design applies. Report standard errors. Cluster them when your
+questions come in groups. Compare paired, not unpaired. Do a power analysis.
+
+As of July 2026, essentially zero mainstream eval tools implement any of it.
+
+I do not think that is because it is hard. The methods are standard. I think it is
+because eval tooling is built by product engineers and this is a statistics
+problem, and those are different people. Miller saw it because he is an
+econometrician looking at a field that had reinvented A/B testing without
+noticing.
+
+So I implemented it. The result is [OpenParity](https://github.com/Skits0098/OpenParity),
+which does exactly one thing: you hand it per-case results from two runs, and it
+tells you whether the difference is real.
+
+It does not run your evals. It does not define graders. It never makes a network
+call. That constraint is what lets its entire test suite run in ten seconds with no
+API key, which matters more than it sounds, and I will come back to it.
+
+Three things I learned building it. The second one is the one I did not expect.
+
+## 1. The gate everyone would build rewards you for having a worse eval suite
+
+The obvious upgrade to `assert score > 0.8` is a significance test. Instead of
+comparing to a threshold, ask: is the candidate significantly worse than baseline?
+
+This is better. It is also broken, and the way it is broken is instructive.
+
+A significance test asks "can I reject the hypothesis that nothing changed?" With
+a small suite, you cannot reject anything. There is not enough evidence to reject
+with. So a small suite passes. Always.
+
+Shrink your eval suite and your build goes green.
+
+That is a catastrophic incentive to bake into a regression tool, and it is a
+completely natural thing to build. You would not notice for months, because the
+failure mode is silence: your CI is green, and it is green because you are not
+looking hard enough to see anything.
+
+The fix is to ask the opposite question. Not "is it worse?" but "can I *prove* it
+is close enough?" That is an equivalence test, TOST, and it is standard in
+pharmacology, where "we failed to detect a difference between this generic and the
+real drug" is obviously not the same claim as "this generic works."
+
+Now an underpowered suite *fails*, because it cannot establish anything either.
+The incentive inverts. To get a green build you need a suite capable of supporting
+a claim. You declare a margin, which is your honest statement of what you do not
+care about, and the tool proves the change stays inside it or admits it cannot.
+
+This gives you four verdicts instead of two:
+
+| | |
+|---|---|
+| `equivalent` | Proven within your margin. Ship. |
+| `regressed` | Significantly worse. Block. |
+| `improved` | Significantly better. |
+| `inconclusive` | Your suite cannot answer. **Not a pass.** |
+
+That fourth one is the whole product. Every other tool has two states, so whenever
+the honest answer is "you don't have enough evidence," it has to pick one and lie.
+
+## 2. The obvious way to handle grouped cases is silently wrong
+
+This is the part I got wrong, and I only know I got it wrong because I wrote the
+tests first.
+
+Eval cases are usually not independent. You have five questions about the same
+document, or eight turns in the same scenario, or twelve prompts against the same
+customer record. When cases share a parent, they move together. Treating them as
+independent understates your uncertainty, and the literature puts the
+understatement above 3x.
+
+Everyone who knows this reaches for the same fix: resample whole clusters instead
+of individual cases. The pairs cluster bootstrap. It is the obvious move, it is
+what I planned, and it is what I built.
+
+Then I ran the calibration test. A 95% confidence interval makes a falsifiable
+promise: across repeated experiments, it contains the true value 95% of the time.
+So generate data where you *know* the true answer, build a thousand intervals, and
+count.
+
+Mine contained the truth 89.5% of the time.
+
+Nominal 95%, actual 89.5%. My intervals were too narrow, which means my tool would
+call regressions that were not there, which is the exact failure it exists to
+prevent. If I had shipped on "the method is standard and my code looks right," I
+would have shipped a tool that lies.
+
+So I measured coverage against the number of clusters:
+
+```
+Continuous data, nominal 95%:
+
+    clusters      pairs bootstrap      CRVE + t(G-1)
+           6               0.845              0.945
+          12               0.902              0.945
+          20               0.917              0.937
+          30               0.923              0.947
+          50               0.950              0.960
+```
+
+And on binary pass/fail data, which is what most eval suites actually produce:
+
+```
+    clusters      pairs bootstrap      CRVE + t(G-1)
+           5               0.840              0.953
+           8               0.890              0.949
+          12               0.931              0.960
+          20               0.948              0.958
+```
+
+This is the documented "few clusters" problem, and there is the shape of it. The
+pairs cluster bootstrap needs roughly fifty clusters before it delivers the
+coverage it advertises.
+
+**Real eval suites do not have fifty clusters.** Group your cases by source
+document and you have eight. At eight, the method quietly gives you 89% coverage
+and prints "95%" on the label.
+
+The fix is a cluster-robust standard error with a t(G-1) critical value, which
+holds ~95% down to five clusters. What does the work there is not the robust
+variance, it is the t correction: at six clusters, t(5) puts the critical value at
+2.57 instead of 1.96, and that gap is the entire difference between 84% and 95%
+actual coverage.
+
+I also measured a wild cluster bootstrap with Rademacher weights. It is
+statistically indistinguishable from CRVE here. CRVE won on engineering grounds:
+a third of the code, analytic rather than resampled, and deterministic, so CI can
+never see the same data produce different bounds.
+
+The general lesson is not about clusters. It is that statistical code fails
+silently. It does not throw. It does not crash. It returns a plausible number with
+a confidence interval attached, and the interval is a lie, and nothing anywhere
+tells you. The only defense is to generate data where you know the answer and
+check that your tool finds it.
+
+Which is why the "no network calls" constraint earns its keep. Because the core is
+pure functions over arrays, I can run ten thousand simulated comparisons in
+seconds for free, on every commit. An eval tool that needs a model to test itself
+cannot do this, and so mostly does not.
+
+## 3. You cannot compute power from one run
+
+I wanted a command that says "your suite is too small, here is how many cases you
+need." Everyone wants this.
+
+You cannot build it from a baseline run. The power of a paired test depends on the
+standard deviation of the per-case *differences*, and that depends on the
+correlation between two runs. One run does not contain that information. Any tool
+that offers power analysis from a single eval set is either guessing sigma or
+silently assuming your runs are independent, which they are not.
+
+What it needs is a **null pair**: run your baseline twice, changing nothing. Same
+prompt, same model, same config. The only difference is sampling randomness. That
+measures your actual noise floor.
+
+This turns out to be the cheapest useful thing you can do to an eval suite. No
+labelling, no judging, no new cases. Just run it twice. And almost nobody has.
+
+Here is what it said about that 50-case suite:
+
+```
+suite:     50 cases
+noise:     SE of the mean difference = 0.0896 (+/-10%, from your null pair)
+goal:      prove equivalence within +/-0.0300 at alpha=0.05
+power:     0.0%  (target 80%)
+
+NOT adequately powered. You need about 3822 cases (+3772) to reach 80% power.
+```
+
+Three thousand eight hundred cases. That number is brutal, and it is not a bug.
+Proving a change stays within three points when your run-to-run noise is nine
+points genuinely takes thousands of cases. That is arithmetic, not pessimism.
+
+The useful response is not to go label 3,772 cases. It is one of:
+
+- **Widen the margin.** Three points was probably a number you picked because it
+  sounded rigorous, not because you can tell the difference between a 2% and a 4%
+  regression in production. Pick the margin you actually care about.
+- **Cut the noise floor.** Sample each case several times and average. This costs
+  inference, not labelling, and it is usually far cheaper than new cases. It also
+  converts binary outcomes into continuous ones, which carry strictly more
+  information per case.
+- **Accept that you are shipping on judgment.** Which is fine. It is what you are
+  doing right now. The difference is knowing it.
+
+What you should not do is keep making ship decisions from a 50-case suite and
+believe them.
+
+## The uncomfortable part
+
+If you run this on your own eval suite, the first thing it will tell you is that
+your suite cannot resolve the question you have been asking it for months. Most
+suites return `INCONCLUSIVE` on their first real comparison. That is the tool
+working, and it is not a pleasant experience.
+
+It also means some of the prompt changes you shipped were noise you read as
+signal, and some you reverted were improvements you threw away. There is no way to
+find that out that feels good.
+
+But the alternative is what everyone is doing now: eyeballing a handful of outputs
+and calling it engineering, while a dashboard reports 78% with three significant
+figures and no error bar. The number is not wrong. It is just not evidence, and
+the two are easy to confuse when one of them is rendered in a nice font.
+
+## Limits, honestly
+
+- **It does not run your evals.** It consumes per-case results from whatever
+  produced them. That is on purpose, so it composes with your existing tooling
+  instead of replacing it, but it means you have to get results out of your runner
+  in a format it can read.
+- **It cannot fix a bad eval.** If your judge is miscalibrated, OpenParity will
+  give you a beautifully rigorous confidence interval around a meaningless number.
+  Statistics do not rescue construct validity.
+- **The power estimate inherits your noise floor's error.** The relative standard
+  error of an SE estimate is about `1/sqrt(2n)`, so about 16% from a 20-case null
+  pair. "You need 355 cases" from a small null pair means "a few hundred," not
+  three significant figures. The tool says so rather than printing false precision.
+- **Below about five clusters, everything is doing heavy lifting.** Coverage held
+  in simulation, but the intervals get wide fast and lean hard on the t
+  correction. Treat borderline verdicts with suspicion.
+
+## Try it
+
+```bash
+pip install openparity
+openparity compare runs/main.json runs/pr.json --margin 0.03
+```
+
+Exit 0 to ship, 1 to block, 2 for "your suite cannot answer this," which also
+blocks, because treating no evidence as success is the failure the whole thing
+exists to prevent.
+
+Source and the full design notes, including the decisions I got wrong first:
+[github.com/Skits0098/OpenParity](https://github.com/Skits0098/OpenParity)
+
+The statistics are verified rather than asserted. Coverage, false alarm rates,
+clustering, pairing, and power are each checked against data whose true answer is
+known by construction, on every commit. If those tests fail, everything above is
+worthless. They are the first thing I would look at, and you should too.
+
+---
+
+### On how this was built
+
+This was written with heavy AI assistance, in one session, using Claude. Every
+commit carries a `Co-Authored-By` trailer, and you can check that against the
+history rather than take my word for it.
+
+I want to be precise about what that does and does not mean, because "AI wrote it"
+is doing a lot of unearned work in both directions right now.
+
+The code was largely generated. What I chose was the scope, and on this project
+the scope was most of the work. This started as something much larger and much
+worse: an open-source "AI workflow control plane" with a dashboard, a durable
+execution engine, approval gates, and traces. That idea is dead for good reasons.
+The orchestration category is saturated, the human-approval piece is a graveyard
+of near-identical abandoned projects, and a broad platform is now the cheapest
+artifact anyone can generate, which is exactly why it signals nothing. Killing it
+and keeping the one piece nobody had built was the decision that mattered.
+
+The other choice worth naming is writing the tests before the implementation, and
+it paid off in a way I did not predict. The cluster bug in section 2 was not
+caught by review or by reasoning. It was caught because a simulation made a
+falsifiable claim and the claim failed. Reading the code would never have found
+it: the code was *correct* for the method it implemented. The method was wrong for
+the data, and the only thing that can tell you that is measurement.
+
+Which is the honest summary of what AI assistance is and is not good for here. It
+will write you a clean, well-documented, confidently wrong statistical library.
+It will not tell you the method is wrong, because it does not know either. The
+simulation knows.
+
+If you want to evaluate this repo, do not evaluate the prose. Run
+`pytest tests/test_calibration.py`, then go break something in `inference.py` and
+watch what happens.