Statistical inference for model inference. Tells you whether an eval delta is real.
You change a prompt. Your eval suite goes from 82% to 78%.
Did you regress, or did you move noise?
No mainstream eval tool can answer this. They compute a mean and compare it to a threshold. On a 50-case suite that comparison is close to meaningless, and it is the reason eval CI is flaky: the tool reports a regression with total confidence when it is looking at sampling error.
OpenParity does the statistics properly and gives you one of four answers:
| Verdict | Meaning |
|---|---|
equivalent |
Proven within your margin. Ship it. |
regressed |
Significantly worse. Block it. |
improved |
Significantly better. |
inconclusive |
Your suite cannot answer this. Not a pass. |
That fourth verdict is the product. Every other tool has two states, so it has to lie whenever the honest answer is "you don't have enough evidence."
pip install openparityfrom openparity import Run, compare
base = Run.load("runs/main.json")
cand = Run.load("runs/pr-142.json")
r = compare(base, cand, margin=0.03, seed=0)
print(r.explain())verdict: INCONCLUSIVE
effect: -0.0400 (0.8200 -> 0.7800)
95% CI: [-0.1600, +0.0000]
90% CI: [-0.1400, +0.0000] vs margin +/-0.0300
p: 0.625 (exact McNemar)
paired: 50 cases
method: bca bootstrap
This suite cannot resolve the question asked of it. That is a finding about the
suite, not a pass. Run `openparity power` to size it.
Four of your fifty cases moved. The interval spans everything from a 16-point regression to no change at all, and McNemar puts it at p = 0.63. There is no evidence here. Most teams would have reverted the change.
openparity compare runs/main.json runs/pr.json --margin 0.03| Exit code | Verdict | Meaning |
|---|---|---|
0 |
equivalent / improved | Ship. |
1 |
regressed | Block. |
2 |
inconclusive | No evidence either way. Also blocks. |
Code 2 blocking is the deliberate part. Treating "we couldn't tell" as success
is the failure this tool exists to prevent, so opting out takes an explicit
--allow-inconclusive. Expect to see 2 the first time you run it on a real
suite; that is the tool working.
--format json for machines, --format md to paste into a PR comment. Pin
--seed in CI so identical data can never produce different bounds.
The CLI is stdlib argparse. OpenParity's only runtime dependencies are numpy and scipy, so it is cheap to vendor into a CI image.
OpenParity does not run evals, define graders or judges, or call an LLM. It consumes per-case results from whatever produced them.
This is deliberate. It means OpenParity composes with promptfoo, DeepEval, pydantic-evals, Inspect, Langfuse, Phoenix, or your own loop instead of competing with them. It also means the core is pure functions over arrays: deterministic, offline, free to test, and provable by simulation.
A 95% confidence interval makes a falsifiable promise: across repeated experiments it contains the true value 95% of the time. Most statistical code in this space has never been checked against that promise.
tests/test_calibration.py checks it, against data whose true effect is known by
construction:
- Coverage. 95% intervals contain the true effect ~95% of the time, on binary and continuous data, under the null and under a real effect.
- False alarms. Under a true null, OpenParity calls a regression about
alphaof the time. Not more. - Clustering. When cases share a parent, naive intervals are shown to collapse to <85% coverage, and declaring the cluster is shown to restore it.
- Pairing. The paired analysis is verified to be dramatically tighter than the unpaired one on the same data.
- Power. Predicted power is checked against the empirical power of the real
compare()function.
If those fail, the library is lying and this README is worthless. Run them.
1. The gate is an equivalence test, not a significance test.
A significance test asks "can I reject the hypothesis that nothing changed?" With a small suite you can't reject anything, so a small suite always passes. The naive test rewards you for having a worse eval suite. Shrink your suite, get a greener build.
OpenParity gates on TOST instead: can you prove the candidate is within margin
of baseline? Now an underpowered suite fails, because it can't establish anything.
The incentive inverts the right way.
2. Cases that share a parent are not independent.
Several questions per document, several turns per scenario. Treating them as
independent understates your uncertainty, and how badly is not a fixed number.
The design effect is 1 + (m-1) * ICC, where m is your cluster size:
- 8 checks per document at a within-document correlation of 0.25 inflates the
standard error about 1.4x.
examples/content_pipeline.pymeasures exactly that, and shows 1.4x is enough to flip a verdict fromimprovedtoinconclusive. Work out your ownmrather than quoting this one. - Miller reports above 3x on popular evals, which have much larger clusters.
Declare cluster_id and OpenParity switches to a cluster-robust interval with
a t(G-1) correction. Note that the obvious fix, resampling whole clusters, is
the wrong one: it needs ~50 clusters before it delivers the coverage it
promises, and real suites have eight. That is not an opinion, it is measured -
see DESIGN.md section 4a.
3. You cannot compute power from one run.
Power for a paired test depends on the standard deviation of the differences, which depends on the correlation between two runs. One run does not contain it. Any tool offering power analysis from a single eval set is guessing sigma or assuming independence.
So OpenParity asks for a null pair: run your baseline twice with nothing changed. That measures your actual noise floor, which is the number you needed all along and probably never measured.
from openparity import power
rep = power(run_a, run_b, threshold=0.03) # two null runs
print(rep.explain())suite: 50 cases
noise: SE of the mean difference = 0.0896 (+/-10%, from your null pair)
goal: prove equivalence within +/-0.0300 at alpha=0.05
power: 0.0% (target 80%)
NOT adequately powered. You need about 3822 cases (+3772) to reach 80% power.
Yes, that number is brutal, and no, it is not a bug. Proving a change stays
within 3 points when your run-to-run noise is 9 points genuinely takes thousands
of cases. The useful response is not to add 3,772 cases. It is to widen the
margin to something you actually care about, or to cut the noise floor itself by
resampling each case and averaging (n_samples), which is usually far cheaper
than new cases. What you should not do is keep making ship decisions from a
50-case suite and believe them.
The methods are standard experimental statistics. The gap is not that this is hard, it is that eval tooling is built by product engineers rather than statisticians.
- Evan Miller, Adding Error Bars to Evals (arXiv:2411.00640, Anthropic). The source. OpenParity is an implementation of its recommendations.
- Hamel Husain, LLM Evals FAQ. Argues binary pass/fail over Likert, partly on statistical grounds.
- Anthropic, Demystifying evals for AI agents. 20-50 real-failure cases as a starting suite; regression evals sit near 100% and exist to detect backsliding.
- Most of your eval regressions aren't real
- why a 50-case suite cannot answer the question you are asking it, and what the simulations caught that code review would not have.
- DESIGN.md - the full design, and decision records for the two choices I got wrong first.
Pre-release, v0.1 in progress.
MIT