Skip to content

CHANGELOG.md

Latest commit

 

History

History
200 lines (184 loc) · 11.2 KB

CHANGELOG.md

File metadata and controls

200 lines (184 loc) · 11.2 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is loosely based on Keep a Changelog, and this project adheres to Semantic Versioning.

Pre-0.3.0 changes are not enumerated here; consult git log for that history.

[0.3.1] — 2026-05-22

Fixed

  • sdist no longer includes maintainer-local working files. Hatchling's default sdist target picked up any non-gitignored files in the project root; pre-0.3.1 the .idea/, RESEARCH.md, and adversarial-loop/ directories were all bundled into the published tarball. Adds an explicit [tool.hatch.build.targets.sdist].exclude list. The wheel was always clean (only benchstone/ source under [tool.hatch.build.targets.wheel]), so this affects sdist consumers only. Tagging the v0.3.0 commit early meant the fix landed in v0.3.1; v0.3.0 was never published to PyPI.

[0.3.0] — 2026-05-22

This release closes the six CONFIRMED findings from the first adversarial-review iteration against benchstone's integrity-claim cluster, plus three deferred quick-wins from the same iteration and the structural keystone fix that subsumes three more. Several are breaking changes; the minor version bump reflects that.

Added

  • STALE verdict kind and a tier-orthogonal freshness guard on bench evaluate / api.evaluate. The guard checks, in order: the working tree's git status --porcelain (dirty?), the manifest's corpus_hash against the on-disk corpus, and a new working_tree_hash against the most recent candidate run at the current SHA. Mismatch on any check returns Verdict(kind="STALE", category="freshness") with exit code 5. Pass --allow-dirty (CLI) / allow_dirty=True (API) to bypass. Skipped for correctness-tier benchmarks.
  • bench evaluate --allow-dirty flag, mirroring bench run --allow-dirty.
  • project.tracked_paths manifest field (default ["bench/"]). Declares the project-relative paths whose byte content is part of the benchmark's input surface. Hashed by the runner on each dispatch, re-hashed by the freshness guard at evaluate time. Projects that import from src/ (or anywhere else) should extend the list.
  • benchstone.tree_hash module with working_tree_hash(project_path, tracked_paths) -> "sha256:...". Deterministic recursive hash; excludes well-known byproducts (__pycache__/, *.pyc, *.pyo, .pytest_cache/, .mypy_cache/, .ruff_cache/, .DS_Store) and symlinks.
  • benchstone.api.check_freshness as a public helper for external callers (custom dashboards, alternate CLIs).
  • runs.working_tree_hash column. Nullable; pre-existing rows are treated as "no hash to compare against" by the freshness guard.
  • baselines.run_ids column (JSON array of run IDs). The baseline pointer now records the exact set of runs that constitute its snapshot, not a query specification that future inserts can expand. See "Changed" → baseline snapshot semantics.
  • Store.fetch_runs_by_ids(run_ids) method — primary read path for baseline snapshots.

Changed

  • Baseline pointer records a run-ID snapshot (the v0.3.0 keystone). Baseline.run_ids is the authoritative selector for the baseline distribution. compute_verdict reads runs by ID, not by (git_sha, meta_seed) query. Closes CONFIRMED-5 structurally (subsequent inserts at the same SHA cannot expand the baseline), closes F-PROMO-FLOOR (promoted-group size disconnect from manifest's baseline_seeds), closes F-MANIFEST-EVOLVE (editing the manifest's baseline_seeds no longer invalidates historical baselines), and closes F-META-RERUN (re-running with the same meta_seed inserts new rows but the snapshot stays fixed). Legacy baselines (pre-0.3.0, run_ids=NULL) fall back to the query-based lookup unchanged.
  • Gate baseline floor is now hardcoded 2 (the math minimum for a sample variance), down from max(len(benchmark.baseline_seeds), 2). Sample-size intent is recorded in the snapshot at establish/promote time; the gate doesn't re-derive it from the live manifest. Strictly more permissive than the old behavior for any sample with ≥ 2 ok rows. Candidate floor unchanged (max(repetitions, 2)).
  • Append AND git_dirty=0 to both branches of Store.fetch_baseline_runs. Dirty rows remain in the timeline (fetch_runs is unchanged) — they no longer count as baseline. This closes the INV-05 baseline-poisoning vector where an agent with --allow-dirty access could inject additional meta_seed=NULL rows at the baseline SHA and shift the gate's view of baseline. With the keystone snapshot, this filter is also redundant for non-legacy baselines (the snapshot doesn't include the injected rows in the first place), but it remains as defense-in-depth and as the active protection for the legacy-fallback path.
  • api.compute_verdict gains optional candidate_meta_seed: int | None = None. Default behavior unchanged (still "latest meta_seed group at this SHA"). Non-None pins the candidate fetch to that group.
  • api.promote and bench promote evaluate the gate against the chosen meta_seed group before moving the baseline pointer, then record the chosen group's run IDs as the new baseline snapshot. Previously, promote --meta-seed N validated only that group N existed and moved the pointer using whichever group was latest at the SHA — letting an explicit --meta-seed ride on a different group's PROMOTE verdict.
  • api.check_freshness signature changed to take project: Project (new first arg, for tracked_paths) and an optional store: Store | None = None kwarg. The function was added in the same release cycle so no realistic external caller exists yet.
  • references.freeze / references.replace ordering inverted — history event is appended before the reference JSON is atomically written. If _write fails after the history append, get() walks history newest-first on every read and raises ReferenceError when the most-recent install event's content_hash disagrees with the live JSON's. Catches the crash-mid-write half-state.

Fixed

  • CONFIRMED-1: bench evaluate / api.evaluate / compute_verdict now refuse with STALE when the working tree is dirty, the corpus has drifted from corpus_hash, or the tracked_paths byte content has changed since the most recent candidate run. Previously emitted the gate's verdict against stored runs without checking the on-disk state.
  • CONFIRMED-2: gitignored modifications under tracked_paths no longer slip past the freshness check. The legacy git status --porcelain view (which omits gitignored entries) is now complemented by the byte-level working_tree_hash comparison, so payloads written to .venv/, __pycache__/, or any project-gitignored path under tracked_paths produce STALE.
  • CONFIRMED-3: promote --meta-seed N now re-evaluates the gate against group N before moving the baseline pointer. Previously a REJECT group could become baseline via the verdict of a separate PROMOTE group inserted later at the same SHA.
  • CONFIRMED-4: bench run --seed-set baseline and bench baseline establish are refused for correctness-tier benchmarks (the invocations produced rows with meta_seed=NULL that fetch_candidate_runs couldn't see, leaving the artifact invisible to evaluate and the default freeze-reference resolution).
  • CONFIRMED-5: baseline poisoning via --seed-set baseline --allow-dirty at the baseline SHA. Closed structurally by the run-ID snapshot (the gate doesn't read additional rows beyond the snapshot) and defense-in-depth by the dirty-row filter.
  • CONFIRMED-6: untracked-only dirty trees no longer have effectively-no audit content. The working_tree_hash is itself the audit primitive for "did the relevant surface change since the run?"; the existing git diff HEAD capture remains unchanged for the human-inspection use case.
  • F-NONFINITE: ProjectResult.from_json rejects NaN / ±Infinity metrics at the protocol boundary. The runner's existing ProtocolError handler converts the violation into a loud status='error' row instead of silently shrinking the sample (NaN → SQL NULL while status='ok') or producing nan/inf statistics (Infinity → mean_se → directed_z).
  • F-REF-ATOMIC: references.freeze and references.replace use history-first ordering (see "Changed"), and references.get raises ReferenceError when the live reference's content_hash doesn't match the most recent history event's recorded hash.
  • F-REF-REVALIDATE: references._require_artifact rehashes the archived bytes and compares to run.artifact_hash before freeze / replace. Catches out-of-band corruption (bit rot, manual tamper, cleanup script) of the content-addressed store.
  • F-PROMO-FLOOR / F-MANIFEST-EVOLVE / F-META-RERUN: subsumed by the run-ID snapshot keystone (see "Changed"). The baseline's sample-size is determined at establish/promote time, not re-derived from a baseline_seeds query at evaluate time.

Breaking changes

  • New exit code 5 for STALE. Consumer scripts that previously only branched on 0/1/2/3/4 should add a 5 case (refuse, ask the user, re-run, whatever the appropriate response is for the consumer). Default bench evaluate behavior on a clean tree is unchanged.
  • bench run --seed-set baseline on correctness-tier benchmarks now raises RunnerError. No callers in the test suite used this combination; external callers (if any) need to switch to --seed-set fresh (the default).
  • bench baseline establish on correctness-tier benchmarks now raises RunnerError for the same reason. Use bench run + bench freeze-reference instead.
  • promote --meta-seed N now refuses when group N's standalone verdict isn't PROMOTE. Previously succeeded silently using a different group's verdict. Pass --force to bypass the gate (same as before, just now actually applied to the chosen group).
  • api.check_freshness signature changed (see "Changed"). Added in the same release cycle; no production callers expected yet.
  • Schema migration: two new columns — runs.working_tree_hash TEXT and baselines.run_ids TEXT. The migration runs automatically on first Store.__init__ against an existing database. Pre-existing rows get NULL; both code paths have documented fallback behavior for that.
  • Gate baseline floor dropped from max(len(baseline_seeds), 2) to a hardcoded 2. Strictly more permissive than before; consumer scripts that branched on NEEDS_MORE_DATA from baseline-undersize may see PROMOTE/REJECT where they previously saw NEEDS_MORE_DATA. Intentional — the prior floor was a footgun (manifest evolution would brick gating).
  • Manifest schema: project.tracked_paths field added. Backward-compatible (default ["bench/"]); manifests that omit it behave identically to before for projects with a bench/ directory.
  • References JSON write ordering inverted (history first, then reference JSON). Old half-state recovery scripts (if any) that assumed write-first ordering need to flip their assumption.