Skip to content

QUALITY_TEST.md

Latest commit

 

History

History
591 lines (426 loc) · 48 KB

QUALITY_TEST.md

File metadata and controls

591 lines (426 loc) · 48 KB
Raw

Quality testing on club-3090

Operational tests (verify / verify-full / verify-stress / bench / soak-test) tell you whether a compose serves correctly. They don't tell you whether the model behaves correctly — whether tool calls land on the right functions, whether instruction-follow constraints hold, whether structured-output stays valid JSON. A compose can pass every operational layer and still ship with degraded behavioral quality from quantization drift, a sampler mismatch, or an engine pin bump.

scripts/quality-test.sh closes this gap. It wraps benchlocal-cli — a CLI port of BenchLocal bench packs — and runs verifier-backed scenarios against the running compose endpoint.

Where it sits in the pipeline

verify.sh         — fast smoke (15s,        "does it serve")
verify-full.sh    — functional (1-2min,     "does everything work")
verify-stress.sh  — boundary (5-10min,      "does it survive stress")
bench.sh          — throughput (3-5min,     "what's the TPS")
quality-test.sh   — behavioral (10-90min,   "does it produce useful output")  ← THIS
soak-test.sh      — stability (30-60min,    "does it stay healthy over time")

Each layer has a different question. Quality testing is the one that catches "passed every other gate but produces wrong tool calls or violates format constraints."

What the packs measure

Five deterministic packs (verifier-backed, no LLM-as-judge — these run without Docker):

Pack Dimension Why it matters for club-3090 users
ToolCall-15 Tool selection + argument correctness IDE-agent traffic (Cline / OpenCode / Cursor) is 100% tool calls. Server-side request rewriting (tool_choice overrides, injected reminder text) causes silent tool-call degradation that operational tests never see
InstructFollow-15 Constraint-heavy instruction compliance Catches "ignore the format constraint" drift from cudagraph mode changes or sampling tweaks.
StructOutput-15 JSON / YAML / markdown structure validity Bounded-thinking, JSON tool args, FSM-constrained reasoning.
ReasonMath-15 Numeric reasoning Code-reasoning correctness; Q4-quant drift surfaces here first.
DataExtract-15 Field-level extraction accuracy RAG / document-Q&A workloads.

Three sandboxed packs add execution-backed verification via Docker sandboxes. They're included in --full (need Docker; --no-sandboxed skips them):

Pack Verifier Why it matters for club-3090 users
BugFind-15 Candidate-fix execution sandbox Code-repair quality + trap-scenario discipline (no false "found a bug").
HermesAgent-20 Multi-tool agent harness (browser / cron / memory / artifact mocks) Multi-step agentic workflows — chained tool calls, recall, delegation. Closest proxy for IDE-agent stacks.
CLI-40 Linux command-exec sandbox Shell/CLI agent tasks (terminal agents like Claude Code / opencode).

A separate eval-expansion pack, AiderPolyglot-30 (multi-language code editing across cpp/go/java/js/python/rust), runs independently — not bundled into --quick/--medium/--full. Drive it via benchlocal-cli run --pack aider-polyglot-30 --enable-sandboxed-packs, or as the aider leg of rebench-full.sh.

The reasoning suite is also separate from --full; run it with --reasoning when you specifically want code/math/science reasoning signal under thinking-on pack defaults:

Pack Verifier Why it matters for club-3090 users
HumanEval+-30 Code execution sandbox over HumanEval+ functional tests Small Python coding tasks; catches code-reasoning regressions quickly.
LiveCodeBench-v6-30 Code execution sandbox over public LCB functional tests Harder post-2025 coding tasks; exposes budget runaway and algorithmic failures.
GSM-Symbolic-30 Deterministic answer_match exact numeric scoring Symbolic grade-school math without LLM-as-judge.
GPQA-Diamond Deterministic answer_match exact letter scoring Science QA placeholder; gated metadata-only until dataset access is materialized, so it reports dataset-unavailable instead of committing restricted data.

Modes

Mode Packs Budget When to run
--quick ToolCall + InstructFollow (2) ~10-15 min Per-commit gate; pre-push smoke. The two packs that catch the highest-value regressions for IDE-agent users. No Docker.
--medium (default) + StructOutput + DataExtract + ReasonMath (5) ~25-30 min Pre-release; pin bumps; new compose authoring. Generates the Quality: line for the compose schema. No Docker.
--full + BugFind + HermesAgent + CLI (8) ~45-60 min Cross-rig comparison; quality A/B vs another quant. The 3 added packs are Docker-sandboxed — needs Docker.
--reasoning HumanEval+ + LiveCodeBench v6 + GSM-Symbolic + GPQA-Diamond metadata (4) ~30-90+ min Dedicated reasoning/code suite. Thinking defaults on for all 4 packs; HumanEval+ and LCB need Docker.

--full runs the sandbox packs by default. --no-sandboxed drops --full back to the 5-pack deterministic scope (no Docker); --sandboxed-only runs just the 3 sandbox packs. --reasoning is independent of --full; use it for the four reasoning packs, with GPQA skipped until gated data is available.

⚠️ --full needs the sandbox images built first — "needs Docker" isn't enough. The 3 sandboxed packs run inside pre-built benchlocal-sandbox-* Docker images that are not auto-pulled, and the build tooling is not in the pip install — it lives in a benchlocal-cli checkout. Build them once:

git clone https://github.com/noonghunna/benchlocal-cli
bash benchlocal-cli/tools/build-sandboxes.sh        # ~30 GB free; the aider image is biggest — `docker system prune` if tight

Then --full works. Without the images, quality-test.sh warns up front and runs the deterministic packs only; for a clean no-Docker run use --medium (or --no-sandboxed). (club-3090 #492 — benchlocal-cli's own mid-run hint pointed at a relative path that's wrong outside a checkout.)

Install (one-time)

pip install git+https://github.com/noonghunna/benchlocal-cli.git

Or for development from a local clone of benchlocal-cli:

pip install -e /path/to/benchlocal-cli

Run

Live progress is on by default. The wrapper forwards --progress to benchlocal-cli, so per-scenario [N/M] <pack> <id> … lines stream to stderr as the run advances. Long modes (--full ~30–40 min, --reasoning similar, --pack aider-polyglot-30 ~25–30 min) otherwise go dark for the whole duration with no signal whether anything is wrong mid-run. Pass --no-progress (or PROGRESS=0) for CI / log-volume-sensitive contexts.

Pass-through (--) and promoted flags

The wrapper names the flags it adds logic to (modes, pack sets, thinking, sampling, timeout). Everything else benchlocal-cli supports goes after -- and is forwarded verbatim — appended after the wrapper's own args, so a pass-through flag can override a wrapper one:

# any benchlocal-cli run flag the wrapper doesn't name:
bash scripts/quality-test.sh --full --no-thinking -- --model-turn-timeout 900 --timeout-ceiling-s 1200

Three of benchlocal's flags are promoted to first-class wrapper flags (help text + validation), per maintainer decision in #1023/#987:

Flag Default What it does
--retry-runaways off (opt-in) Also retry timeout / token_limit runaway failures. benchlocal retries model verdicts 3× but never runaways — each attempt is a full generation. Opt in when slow-rig timeout rows (a single sample against a clock, not a model verdict) are polluting the score.
--strict-thinking off Exit code 4 when the thinking-validity check finds a contaminated arm (e.g. the "no-thinking" leg secretly reasoned). CI-friendly; pairs with the canonical two-leg run above.
--report FMT + --report-out PATH off Emit benchlocal's paste-ready Results Card v2 report instead of relying on the compact one-liner: --report md --report-out card.md.

The drift is guarded: every flag named for quality-test.sh in this doc or CLAUDE.md must be either forwarded by the wrapper or appear after -- (scripts/tests/test-quality-passthrough.sh). If you document a new flag in a wrapper invocation, wire it or route it through --.

⭐ The canonical two-leg run

This is the recipe every announcement quotes and the one to copy if you're producing a number anyone else will read. Substitute your slug.

⚠️ Each leg needs its OWN boot, in both directions. The container persists between runs, so going back to instruct after a thinking run needs a reboot too. Skip it and the "instruct" leg is silently a second thinking leg — both arms score alike and the A/B reads as a clean null. benchlocal flags it: per-pack thinking_validity in the saved JSON, or --strict-thinking for a CI exit code.

# ---- leg A: instruct (the shipped default — no env vars) ----
bash scripts/switch.sh --force vllm/qwen38-27b-dual-fast
bash scripts/quality-test.sh --full --no-thinking --sampling-from-server \
  --max-tokens 4096 --thinking-max-tokens 16384 --timeout-per-case 600

# ---- leg B: thinking (one var — the compose derives the card's thinking
#      sampler from it) ----
ENABLE_THINKING=true bash scripts/switch.sh --force vllm/qwen38-27b-dual-fast
REASONING_EFFORT=low BENCHLOCAL_MODEL_TURN_TIMEOUT=900 \
bash scripts/quality-test.sh --full --enable-thinking --sampling-from-server \
  --max-tokens 4096 --thinking-max-tokens 16384 --timeout-per-case 600

ENABLE_THINKING now flips BOTH the chat template and the sampler on models with per-mode card rows (qwen3.8-27b): the compose entrypoint picks the matching model-card row (:= defaults, so an explicit TEMP/TOP_P/PRESENCE_PENALTY still wins). The old four-variable ritual is obsolete — #1014, guarded by scripts/tests/test-compose-sampler-profiles.sh. Single-row models are unaffected: there, setting the three sampler vars by hand remains the only way to change them.

Why each flag — none is decoration; each exists because its absence produced a wrong number:

Flag Why
--sampling-from-server Uses the compose's card-correct sampler instead of the pack's temperature=0. Without it you measure greedy decoding, not the shipped config.
--max-tokens 4096 The ~1024 default silently truncates long answers into token_limit failures that look like wrong answers.
--thinking-max-tokens 16384 The thinking arm needs the headroom.
--timeout-per-case 600 16,384 tokens takes real wall-clock. Too tight a cap produces timeout rows that read as content misses.
REASONING_EFFORT= Env only — forwarded as the per-request OpenAI reasoning_effort. The qwen3.8 composes default to low server-side since #1029; pin it anyway so the run records what it measured. ⚠️ Effort is not comparable across runs. ⚠️ high raises on vLLM — only xhigh/medium/low exist.
--repeat 3 Add it for anything you'll quote. With --sampling-from-server both legs are sampled, so single draws aren't quotable.

⚠️⚠️ Read the failure modes before reading the score. timeout and token_limit are harness artifacts; only verifier_fail / wrong_answer are the model. A score that looks catastrophic is usually a budget that was too small. And benchlocal retries model verdicts 3× by default but does not retry timeouts — so a timeout row is a single sample against a clock. Opt into runaway retries with --retry-runaways (default off; each attempt is a full generation) when the clock, not the model, is the suspect.

Operational health is a separate pass — don't conflate them

Behavioural quality (above) and operational health split cleanly. Run one of each and you've covered everything with nothing run twice:

bash scripts/report.sh --full     # ~43 min: verify + stress + soak + bench + agentic, redacted

⚠️ Don't pair rebench-full.sh with report.sh --full — rebench re-runs the same operational gates, so it replaces that command rather than complementing it. Pick by goal: rebench-full --with-8pack-thinking=both for one synthesized REPORT.md (quant A/B, BENCHMARKS row); the two-pass split above for the paste-ready cross-rig bundle.

⚠️ Long runs: redirect to a file, never pipe through tail/head/grep. A pipe buffers until the process exits, so a --full run piped to tail is a ~2 h black box with no live [N/M] progress and nothing readable if you interrupt it. Do bash scripts/quality-test.sh --full … > run.log 2>&1 and tail -f the file.

Other invocations

# default --medium against the auto-detected running compose
bash scripts/quality-test.sh

# faster mode (per-commit gate)
bash scripts/quality-test.sh --quick

# full mode (pin bumps, cross-rig comparison)
bash scripts/quality-test.sh --full

# dedicated reasoning suite (thinking-on pack defaults; code packs need Docker)
bash scripts/quality-test.sh --reasoning

# explicit endpoint override
URL=http://localhost:8011 bash scripts/quality-test.sh --quick

# suppress live [N/M] progress for CI / log-volume contexts
bash scripts/quality-test.sh --full --no-progress

# --full includes the 3 Docker-sandboxed packs by default (BugFind/HermesAgent/CLI) — needs Docker
bash scripts/quality-test.sh --full

# skip the sandbox packs (drops --full to the 5-pack deterministic scope, no Docker)
bash scripts/quality-test.sh --full --no-sandboxed

# run ONLY the 3 sandbox packs
bash scripts/quality-test.sh --sandboxed-only

# run individual reasoning packs
bash scripts/quality-test.sh --pack humaneval-plus-30 --enable-thinking --thinking-max-tokens 16384 --timeout-per-case 300
bash scripts/quality-test.sh --pack lcb-v6-30 --enable-thinking --thinking-max-tokens 16384 --timeout-per-case 300
bash scripts/quality-test.sh --pack gsm-symbolic-30
bash scripts/quality-test.sh --pack gpqa-diamond

Output:

  1. Markdown table to stdout — paste-ready for BENCHMARKS quality rows
  2. JSON to results/quality/quality-<timestamp>.json — full per-scenario detail for delta tracking
  3. One-liner suitable for the compose Quality: schema field — paste into compose YAML header

Example output:

=== benchlocal-cli --medium  (endpoint: http://localhost:8020, model: qwen3.6-27b) ===

Pack                       | Pass / Total | Score | p50 latency | p95 latency | Status
ToolCall-15 (v1.0.1)       |   14 / 15    |  93%  |     8.2s    |     12.1s   | ✅
InstructFollow-15 (v1.0.0) |   13 / 15    |  87%  |    11.4s    |     17.8s   | ✅
StructOutput-15 (v1.0.0)   |   15 / 15    | 100%  |     6.9s    |      9.2s   | ✅
DataExtract-15 (v1.0.0)    |   12 / 15    |  80%  |     7.3s    |     10.5s   | ✅
ReasonMath-15 (v1.0.0)     |   11 / 15    |  73%  |    14.2s    |     22.6s   | ✅
─────────────────────────|──────────────|───────|─────────────|─────────────|──────
TOTAL                      |   65 / 75    |  87%  |             |             |

Failure breakdown:
- toolcall-15 TC-07: verifier_fail (wrong arg value for "filename": expected report.pdf, got output.pdf)
- instructfollow-15 IF-03: verifier_fail (word count 247, target 250 ±5)
- dataextract-15 DE-05: verifier_fail (7/14 atomic fields correct (50%). product_name: mismatch | product_price_paid: expected number)
- reasonmath-15 RM-09: verifier_fail (expected 42, got 45)

==========================================================================
Quality: line for compose schema field (paste into compose YAML header):
==========================================================================
Quality:   ToolCall-15 14/15 (93%) · InstructFollow-15 13/15 (87%) · StructOutput-15 15/15 (100%) · DataExtract-15 12/15 (80%) · ReasonMath-15 11/15 (73%) (--medium, thinking OFF, sampling=server, validity=valid, packs tc1.0.1·if1.0.0·so1.1.0·de1.2.0·rm1.0.0, 2026-05-09)

Cloud / proxy endpoints

quality-test.sh runs the same 8-pack against any cloud or proxied OpenAI-compatible endpoint — a managed API (DashScope, OpenRouter, together, DeepInfra), a router, or your own hosted model — for a like-for-like local-vs-cloud comparison (identical prompts, identical verifiers). Cloud support shipped in #746; the underlying knobs live in benchlocal-cli → Running against a cloud / managed endpoint.

Three env vars (or flags) point the wrapper at a remote endpoint:

Knob Flag Purpose
URL --endpoint The endpoint's OpenAI-compatible base (https://…/v1).
API_KEY --api-key Bearer token (Authorization: Bearer <key>); falls back to BENCHLOCAL_API_KEY.
MODEL --model The model id the endpoint serves (overrides the local auto-detect default). 
# thinking-ON arm against a cloud endpoint
URL=https://your-endpoint/v1 API_KEY="$YOUR_KEY" MODEL=your-model-id \
  bash scripts/quality-test.sh --full --enable-thinking --incremental --save-json cloud-on.json
# thinking-OFF arm
URL=https://your-endpoint/v1 API_KEY="$YOUR_KEY" MODEL=your-model-id \
  bash scripts/quality-test.sh --full --no-thinking --incremental --save-json cloud-off.json

Reachability probe is cloud-tolerant. Proxies (a LiteLLM master key) answer 401 without the key, and some cloud endpoints (DashScope MaaS) serve only /chat/completions and 404 on /v1/models. The wrapper treats any HTTP response as reachable (only http_code 000 = no connection fails fast), so cloud/proxy references pass the preflight without a /v1/models route.

Match the thinking state explicitly. Most managed endpoints ignore the vLLM-side chat_template_kwargs.enable_thinking field — use the provider's native controls. The cli-40 / hermesagent-20 adapters send Qwen-compatible enable_thinking + thinking_budget automatically; for a thinking-only endpoint the off arm is enable_thinking=true clamped to thinking_budget=1. DashScope rejects enable_thinking=false, so its off arm uses that clamp (worked example below). Run both arms for a fair comparison and verify the saved request payloads.

Pacing + spend. Pass -- --request-delay <sec> (env BENCHLOCAL_REQUEST_DELAY) to stay under the endpoint's RPM ceiling, and -- --max-total-tokens <N> as a cost ceiling (the agentic packs spend the most) — both are benchlocal-cli flags the wrapper reaches via the -- pass-through. 429s auto-retry (--max-transient-retries, default 3) — see benchlocal-cli #106 for the minute-window backoff caveat.

Worked example — Qwen3.8-Max-Preview (DashScope)

Our first cloud reference (Discussion #753): qwen3.8-max-preview via a LiteLLM proxy that normalizes auth + thinking controls into two routes — qwen3.8-max (thinking-on) and qwen3.8-max-nothink (thinking_budget=1). Result: 125/150 think-off · 134/150 think-on (n=1) — see the cloud references table.

# via the proxy routes (services/litellm/config.yaml)
URL=http://localhost:4000 API_KEY="$LITELLM_KEY" MODEL=qwen3.8-max \
  bash scripts/quality-test.sh --full --enable-thinking --save-json qwen38max-on.json
URL=http://localhost:4000 API_KEY="$LITELLM_KEY" MODEL=qwen3.8-max-nothink \
  bash scripts/quality-test.sh --full --no-thinking --save-json qwen38max-off.json

Scenario-level probes (selection, incremental, resume)

Since benchlocal-cli #84/#85 the wrapper passes through scenario-granular runs:

# one or more specific scenarios (pack-qualified, repeatable)
bash scripts/quality-test.sh --scenario cli-40/CLI-31 --scenario reasonmath-15/RM-04 --no-thinking

# a curated probe set from a file (newline PACK_ID/SCENARIO_ID, # comments),
# BOTH reasoning modes — same pairing as a full eval:
bash scripts/quality-test.sh --scenarios-file scripts/scenario-sets/tess4-model-floor.txt --no-thinking
# ⚠ ON leg: boot the compose with reasoning parsing on FIRST (REASONING=on for
#   llama.cpp composes, --reasoning-parser for vLLM) so <think> lands in
#   reasoning_content, not the graded answer — then:
bash scripts/quality-test.sh --scenarios-file scripts/scenario-sets/tess4-model-floor.txt \
    --enable-thinking --repeat 3

# journal each scored scenario (fsynced sidecar) so an interrupt is resumable
bash scripts/quality-test.sh --full --no-thinking --incremental

# resume an interrupted (or inspect-then-continue) run — restores the original
# pack-set/selection/thinking/sampling/timeout config; only missing arms run
bash scripts/quality-test.sh --resume results/quality/quality-<ts>.json.partial.jsonl

Probe discipline (the tool enforces most of this):

  • A selection result is PARTIAL — the JSON carries top-level selection + per-pack catalog_scenario_count, human output says PARTIAL SELECTION, and history ingestion / rescore refuse it without --allow-partial. It is never a /150 claim — full 8-pack both modes remains the bar for BENCHMARKS rows, Quality: lines, and promotions.
  • Thinking-ON probes sample at temp 1.0 by pack contract → single ON probes are draws; pass --repeat 3 (cheap at scenario granularity) when a number gates a decision.
  • --resume is mutually exclusive with mode/pack/selection/thinking/sampling/timeout flags — it restores those from the saved run; the wrapper refuses the combination rather than fork the config.

Curated probe sets live in scripts/scenario-sets/ with provenance headers:

file what when to run
tess4-model-floor.txt 14 fails-everywhere (+2 thinking-only) across 2 rigs / 2 drafters / 2 engine builds — the Tess retrain-target list (#665 intersection) before/after a Tess fine-tune or retrained drafter head; quantifying a "did the model move" claim. Measured (Tess dual, b9967, 2026-07-12): OFF ~3.5 min · ON ~11 min single draw (ON ×3 ≈ 30 min — still ⅓ of one full 8-pack leg)
tess4-engine-window.txt CLI-25/31/32 — the b9932→b9967 engine-window flips first probe on any new engine build/pin arm, before paying for a full 8-pack. Measured: ~40 s OFF

scripts/rerun-failed-packs.sh now re-runs a prior run's failures as ONE selection run (was: whole-pack loops) — 6 failures over 5 packs = 6 scenarios, with --incremental durability and a REPRODUCED/FIXED verdict per original failure. RERUN_DRY=1 previews the plan.

Which probe for which question — cheap comparison before a full eval

A full 8-pack (both modes) is ~1–2 h and the bar for any published number. But most day-to-day questions — "is this quant better?", "did the retrain move?", "did the engine bump help?" — don't need it. They need the right cheap probe, because the discriminating scenarios depend on what you're comparing, and a targeted probe runs in minutes.

The principle that makes this work: same-family checkpoints tie on the deterministic packs and diverge only on specific fragile ones. So you don't re-measure what won't move — you probe where the difference lives, and only "earn" the full eval when the probe moves.

You're asking… Probe "Better" means Why it discriminates
Is this a better quant / recipe of the same model? cli-40 (--pack cli-40, ~15 min) higher cli-40 (precision preserved) Quant differences concentrate in agentic behavior; deterministic packs (TC/IF/SO/DE/RM) tie across recipes, so cli-40 is where 4-bit-vs-8-bit-vs-GGUF actually separates. Proven 2026-07-12 (recipe arms below).
Did a retrain / new fine-tune of this model line crack its known-hard scenarios? the model's floor set (e.g. scenario-sets/tess4-model-floor.txt, ~15 min) more floor scenarios pass (capability added) The floor is the model's hardest scenarios; quant can't move them (it preserves/degrades, doesn't add capability) — only real training does.
Did an engine build / pin bump help? the engine-window set (e.g. tess4-engine-window.txt, ~40 s) the flip scenarios pass Isolates the handful of scenarios a build version is known to move; the rest are engine-invariant.
Is a new / different model worth a full eval at all? --medium (5 deterministic packs, ~15–25 min) overall lift A different model has its own floor — the Tess floor won't gauge a Qwen. A broad slice is the right first screen.
Are last run's failures real or flaky? scripts/rerun-failed-packs.sh <result.json> REPRODUCED vs FIXED Re-runs only the failed scenarios as one selection.

The gate rule (this is the whole method): a probe is a cheap positive trigger, not a verdict. Probe moves → run the full 8-pack for the real number. Probe doesn't move → you've saved ~2.5 h, and that's the call for quant/engine comparison (they either move the fragile pack or they don't). It is the exact rule the recipe arms used: cli-40-OFF ≥ 21 earned a full run; huginnfork's 18 didn't, FP8's 22 did.

Two rules to not over-apply it:

  • The floor is model-line-specific and a one-way trigger. A retrain that lifts the mid-tier churny scenarios (the ones that pass 1-in-4) can leave the floor flat — so a flat floor is ambiguous, not a "skip." Floor-moved is a strong yes; floor-flat means fall back to --medium or the full run, don't conclude "no gain."
  • OFF is the clean discriminator; ON is churny. Gate on the greedy OFF leg (deterministic, reproducible). ON legs sample at temp 1.0 — a single ON probe is a draw; use --repeat 3 if an ON number is load-bearing.

Free pre-screen where you have it: KLD. If the checkpoints ship KL-divergence self-reports (many quant exports do), they predict the quant ranking at zero GPU cost — 2026-07-12 the reports (fp8 0.013 < NVFP4A16 0.042 < NVFP4-W4A4) called the cli-40 order exactly. Sort by KLD, then cli-40-probe only the top candidate.

Worked example (2026-07-12 recipe arms). Comparing four Tess checkpoints (migtissera NVFP4 / huginnfork NVFP4A16 / FP8 / GGUF) the naive way = four full 8-packs ≈ 10 h. Instead: KLD pre-screened the order, a cli-40 probe (~15 min each) ranked all four and gated the full runs, and only the two that cleared the gate got a full 8-pack. Total ≈ 2 h, same conclusion (precision is the lever, FP8 111/117 ties the GGUF-ON) — see learnings/tess-4-27b.md 2026-07-12 and #662.

pass@1 vs pass@N — the churn-harvest ceiling (and why we don't report it)

Every /150 total in this repo is pass@1 at pack-contract sampling: think-OFF legs are greedy (deterministic), think-ON legs are a single draw at temp 1.0 / top-p 0.95 / top-k 20. That contract is what makes totals comparable across rigs, engines, and dates.

The observation (from the #665 cross-rig work, 2026-07-12): at temp 1.0, many "failing" scenarios aren't failures — they're churners with a per-draw pass probability. Measured examples on Tess-4-27B: scenarios that read as hard-0 on any single run pass 1-in-7 to ~2-in-5 across repeated draws (tess4-model-floor.txt Tier 2 documents six of them with evidence). Take the union of passes across enough draws and the effective ceiling rises sharply: a 7-draw window on a single 4090 reached ~139/150-equivalent coverage, and across every stack we've measured only 10 scenarios sit at p≈0 (Tier 1). The gap between a model's pass@1 total (~116–118) and its churn-harvest ceiling (~139) is ~20 points of probability, not capability.

Two consequences, deliberately kept apart:

1. As a serving technique, harvesting is legitimate — and now cheap to size

If the caller owns a verifier — tests pass, JSON validates against a schema, an archive hash matches, a migration applies cleanly — then verifier-guided best-of-N (rejection sampling) converts probability gaps into successes at predictable cost:

per-draw p N for ≥90% N for ≥99%
0.15 15 29
0.30 7 13
0.40 5 10

(P = 1 − (1−p)^N; cost ≈ N× tokens plus the verifier, and draws parallelize — see the concurrency numbers in FAQ.) Agent harnesses already do a degenerate version of this via retry-on-error; doing it deliberately, with the validator run before accepting, is strictly better. Measuring a scenario's p is now a minutes-scale task: --scenarios-file <set> --repeat N returns per-scenario pass rates directly.

When it applies: only where verification is cheaper than generation and mechanical (schema/tests/hashes). It does nothing for open-ended prose, and nothing for Tier-1 capability gaps — no N rescues p≈0.

What it is not (yet): a stack feature. It's a client-side pattern; if it graduates, it would be a retry-with-validator wrapper in front of the endpoint, never an engine or compose change. Structured-output constrained decoding remains the first choice where the check is expressible as a grammar — best-of-N is the fallback for checks that only a verifier can run.

2. As a benchmark number, harvesting is laundering — and the tooling refuses it

pass@N and pass@1 are different metrics, and mixing them inflates a model's number with the verifier's work. This is why the guardrails are shaped the way they are:

  • Selection results are labeled PARTIAL SELECTION and refuse history/rescore ingestion without --allow-partial.
  • --repeat N aggregates at ≥50% per scenario — a majority vote, not a best-of harvest.
  • Canonical sampling is pinned per pack; overrides mark the run non-canonical.

Reporting rules: BENCHMARKS /150 columns are pass@1-at-contract, always. If you publish a harvested number, label it pass@k with k and the verifier stated (e.g. "pass@7, pack verifiers as oracle") — and never in the same column as pass@1 totals. Scenario-level claims ("X now passes") follow the same discipline: a churner observed once is 1/N draws, not "passes".

Credit: the ceiling observation and the "probability lifted vs capability trained in" framing come from @seanyourhighness's 7-draw b9967 window in #665.

Diagnosing failures

Failure reasons are surfaced in three places, cheapest first:

Need Where
Why a scenario failed (reason + detail), run just finished The Failure breakdown: block at the end of every run — pack scenario: failure_mode (detail), full detail string. No extra command.
Same, but the run scrolled away / an older run results/quality/quality-<ts>.json (raw), or benchlocal-cli inspect <json> --failed
The full prompt / response / verifier trace behind a failure benchlocal-cli inspect <json> --scenario <ID> --full
Filter by failure type · compare two runs · per-scenario tokens + latency benchlocal-cli inspect <json> --mode timeout · --diff prev.json

failure_mode is one of: verifier_fail (answer wrong / below threshold) · timeout · agent_runner_timeout / agent_runner_crashed (sandboxed agentic packs) · server_error / http_error / model_endpoint_unreachable (serving problem, not a quality signal) · result_json_malformed · wrong_answer · verifier_not_implemented (stub, excluded from scoring).

The breakdown is terminal-only — quality-test.sh does not tee it to a log file, but the same data persists in the saved JSON.

Per-scenario timeouts

quality-test.sh forwards to benchlocal-cli, which sizes each scenario's timeout automatically — you rarely need to set one. Precedence (highest wins):

  1. Manual — --timeout-per-case N (or TIMEOUT_PER_CASE=N): used verbatim.
  2. Auto-scaling (default) — the budget scales by the endpoint's measured decode speed and, for thinking-on runs, by the thinking-token budget. A one-shot startup probe measures the rig's decode TPS (and fails fast if the endpoint is unreachable, rather than hanging). The scaling deliberately over-budgets — a timeout is a safety ceiling, not a target — which is what keeps thinking-on packs from spuriously timing out. Exact formula + tuning flags (--measured-tps / --reference-tps / --retry-on-timeout — reach them through the -- pass-through): benchlocal-cli README → Per-case timeouts.
  3. Static default — the pack's built-in default_max_seconds.

Don't hand-set --timeout-per-case to "fix" a slow run unless you've confirmed the auto-probe measured wrong — the over-budget is intentional.

Planned (not yet built): an opt-in tier that sizes timeouts from a soak-derived per-context-depth TPS curve — a real "how fast at depth X" measurement for your exact rig/config, captured into the runtime measurement-record — instead of the single empty-context startup probe. It would be strictly opt-in and fall back to the auto-probe/default when no curve exists; measured data is never required. Tracked at #114.

Sampling & temperature

By default the packs sample at temperature 0 (greedy) — deterministic and reproducible, so scores are comparable across rigs and across runs. This is the canonical baseline, and it's what regression tracking and cross-config ranking should use.

Two opt-in modes evaluate a model at a non-zero / model-recommended temperature instead. Both tag the run non-canonical (markdown header + saved JSON) and refuse to gate CI:

Mode What it does When to use
--sampling-from-server Omits all sampling params from requests, so the server applies its compose-configured defaults; reads them back from /props (llama.cpp) and records them. The compose is the single source of truth. "Evaluate the model exactly as it's served."
benchlocal-cli … --temperature N (+ --top-p / --top-k / --min-p / --repeat-penalty) Eval at sampling values you specify. Mutually exclusive with --sampling-from-server. When you know the model's recommended temp and want it explicit and recorded.

The composes ship model-recommended sampling defaults (Qwen3.6 0.6, Qwopus3.6 0.8, Gemma 1.0), set via the TEMP / TEMPERATURE / TOP_P / TOP_K / MIN_P / REPEAT_PENALTY env (see .env.example). --sampling-from-server inherits whatever the running compose declares — so "serve at the recommended temp" and "eval at the recommended temp" stay in sync from one source.

# canonical (default): temp 0, reproducible — use for ranking + regression tracking
bash scripts/quality-test.sh --full

# evaluate at the model's served / recommended temperature (inherits the compose default)
bash scripts/quality-test.sh --full --sampling-from-server
SAMPLING_FROM_SERVER=1 bash scripts/rebench-full.sh --with-8pack-thinking=both

# or an explicit temperature, via benchlocal-cli directly.
# NB: invoking benchlocal-cli directly BYPASSES the wrapper's localhost guard. With a
# localhost endpoint + a sandboxed *agentic* pack (HermesAgent-20 runs the agent INSIDE
# the sandbox), you must set BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1 yourself — otherwise the
# in-sandbox agent can't reach the host model and hermes silently scores ~0/20.
# quality-test.sh sets this automatically for localhost URLs (see Limitations).
BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1 \
  benchlocal-cli run --full --endpoint http://localhost:8020 --model <name> --temperature 0.8

Reasoning-on evals

Serving with a model's reasoning flag enabled is necessary but not sufficient: the request also has to send chat_template_kwargs.enable_thinking=true. benchlocal-cli honors each pack's default_thinking metadata, so the dedicated --reasoning suite defaults thinking on for all four packs while many format/extraction packs stay answer-only. Use --enable-thinking only when you want to force thinking on for every pack in a broader mode such as --full:

# dedicated reasoning suite; default thinking is on for these packs
bash scripts/quality-test.sh --reasoning --thinking-max-tokens 16384

# force thinking on for every full-suite pack
bash scripts/quality-test.sh --full --enable-thinking --thinking-max-tokens 16384

# full rebench incl. the 8-pack in both reasoning modes (off + on — the promotion gate).
# The 8-pack thinking is driven by --with-8pack-thinking (=off forces --no-thinking,
# =on forces --enable-thinking), NOT ENABLE_THINKING (which now only affects bench.sh). #338
THINKING_MAX_TOKENS=16384 SAMPLING_FROM_SERVER=1 bash scripts/rebench-full.sh --with-8pack-thinking=both

# TPS bench only
ENABLE_THINKING=1 bash scripts/bench.sh

If /props or the running container suggests reasoning is enabled but the wrapper is not forcing thinking on globally, quality-test.sh / bench.sh print a warning; pack defaults still apply, and --enable-thinking forces every pack on. --thinking-max-tokens now passes through independently and only affects packs whose thinking gate resolves on. The default is 16K; hard LiveCodeBench items may still exhaust that budget, so compare with benchlocal-cli run --reasoning --no-thinking when diagnosing budget runaway.

Why it matters: a reasoning / exploratory fine-tune (e.g. Qwopus3.6, whose author recommends temp 0.75–1) is under-represented at temp 0 or with thinking disabled — greedy, thinking-off decoding collapses the path-exploration the fine-tune was trained for. But high temp and reasoning also hurt deterministic packs (DataExtract / StructOutput want exact, repeatable output), so read per-pack deltas, not just the total — and keep canonical temp-0 thinking-off as the bar for any apples-to-apples ranking.

Compose Quality: schema field

Each compose's Profile header (per AGENTS.md) can carry an optional Quality: line:

# Profile (at-a-glance):
#   Model:     Qwen3.6-27B (Lorbus AutoRound INT4 + BF16 mtp.fc preserved)
#   Topology:  Dual 3090 PCIe (TP=2, no NVLink)
#   ...
#   Status:    ✅ Production
#   Quality:   ToolCall-15 14/15 (93%) · InstructFollow-15 13/15 (87%) · StructOutput-15 15/15 (100%) · DataExtract-15 12/15 (80%) (--medium, thinking OFF, sampling=server, validity=valid, packs tc1.0.1·if1.0.0·so1.1.0·de1.2.0·rm1.0.0, 2026-05-09)
#   Best for:  General-purpose dual-card vision + tools + long-ctx default ⭐

The line documents what the compose was tested on — against which pack versions (#981). The same model responses score 4/15 or 9/15 on DataExtract depending only on the pack version, so a number without version provenance is untraceable. quality-test.sh generates the line from the results JSON at end of run: paste it VERBATIM. Each stamp appears only when the JSON carries it:

Stamp Meaning
thinking OFF / ON reasoning gate forced off/on for every pack (absent = pack defaults)
sampling=server --sampling-from-server: sampling inherited from the serving config (absent = canonical pack-default temp=0)
validity=valid / CONTAMINATED #126 thinking-validity check: CONTAMINATED means a requested arm did not reason as asked — do not trust that leg
packs tc1.0.1·if1.0.0·… exact per-pack versions, compact ids (tc=toolcall-15, if=instructfollow-15, so=structoutput-15, de=dataextract-15, rm=reasonmath-15, bf=bugfind-15, hm=hermesagent-20, cli=cli-40)

Never hand-write a packs v1.0.x wildcard — the eight packs span six distinct versions. For richer provenance (per-pack latency p50/p95, variance under --repeat, benchlocal-cli version), pass --report md --report-out card.md and link the generated Results Card v2 next to the Quality line.

Supported two-leg workflow (#983): run thinking OFF and ON with --both-modes, or by hand with --no-thinking --sampling-from-server then --enable-thinking --sampling-from-server. The compose is the single source of truth for per-mode sampler rows (the ENABLE_THINKING=1 env/flag couples the request-level gate to the serving config's sampler block); never pass explicit sampler numbers (--temperature etc.) — they duplicate the model card into a second place that drifts, and leg A vs leg B must differ ONLY in the thinking gate to be a valid A/B.

Compact format (one line) so the schema header doesn't bloat. Full per-scenario detail lives in the JSON saved by quality-test.sh, which can be diffed against past runs for regression tracking.

What "passing" means

quality-test.sh does NOT enforce a hard pass/fail threshold. The script always exits 0 if the runner completes; you decide whether the scores are acceptable.

Suggested gates (informal, not enforced):

Pack Suggested floor Notes
ToolCall-15 ≥80% Below this, IDE-agent users will report regressions
InstructFollow-15 ≥80% Below this, format-constraint workflows break
StructOutput-15 ≥90% JSON shape failures are visible immediately to users
DataExtract-15 ≥75% Slightly more tolerant; field-level scoring is granular
ReasonMath-15 ≥60% Reasoning quality varies more by quant; treat as informational

For comparing a new pin / quant / config A/B against the previous version: a >10pp drop on any pack vs the previous baseline is a signal worth investigating before promoting Status: ✅ Production.

Rescoring saved results — MATERIALIZE, don't just read

When a harness fix changes how saved runs score (e.g. the benchlocal-cli #79/#81 fairness + reasoning-channel fixes), re-score the SAVED result JSONs with the rescore subcommand — and write the corrected results back into the artifact:

benchlocal-cli rescore results/rebench/<tag>/quality-full-thinking.json --in-place

rescore re-runs the deterministic scorers against each run's saved raw_response (sandbox packs like hermes/cli-40 are skipped — those need a live re-run). Printing the corrected totals to stdout and publishing them without --in-place/--output leaves the tag artifact stale — anything that later reads the tag (the catalog-baseline.sh induction tool, rebench-report.py, the measurement-record corpus) silently resurrects the pre-fix numbers. This bit the Agents-A1 gate: the published thinking-on 110/150 was rescore-corrected, but the tag JSON read 108/150 until the rescore was materialized on 2026-07-04.

Rule: a rescore that changes a number you publish must be materialized into the tag artifact in the same session (keep a copy of the pre-rescore JSON elsewhere if you want the history; the tag carries the accepted truth).

Regression baselines (the curated corpus)

Rather than hunt down "the previous baseline" by hand each time, the curated corpus in results/baselines/ holds committed n≥3 aggregates per (registry-slug, thinking-mode). scripts/quality-baseline.sh captures and diffs against them:

# capture / refresh a baseline (n=3 aggregate; needs a live endpoint)
bash scripts/quality-baseline.sh --slug vllm/qwen-35b-a3b-dual --capture
# diff a fresh run vs the committed baseline — the regression check
bash scripts/quality-baseline.sh --slug vllm/qwen-35b-a3b-dual
# thinking-on companion baseline
bash scripts/quality-baseline.sh --slug vllm/qwen-35b-a3b-dual --mode enable-thinking

no-thinking is canonical (temp-0, reproducible — diff against this for a CI-style gate); enable-thinking is the reasoning-on companion. It's a thin wrapper over quality-test.sh --full (--repeat → --save-json/--previous-result); extra args pass through (e.g. --exit-on-regression for a hard CI gate). --dry-run prints the resolved command.

What it doesn't replace

  • bench.sh measures throughput, not quality. They're complementary.
  • soak-test.sh measures stability over time. Quality + soak together catch "fast + correct + healthy."
  • NIAH (needle-in-haystack) tests in verify-stress.sh measure long-context retrieval correctness — a different axis than tool-call / instruction-follow.

Limitations

  1. Sandboxed packs need Docker — BugFind / HermesAgent / CLI-40 run in Docker-hosted verifier sandboxes. On a host without Docker, run --medium (or --full --no-sandboxed) for the 5 deterministic packs.

  2. Sandboxed agentic packs need a container-reachable model URL — HermesAgent-20 runs the agent inside the sandbox, so it calls the model over the network. A localhost / 127.x / [::1] endpoint is the container's own loopback, not the host. quality-test.sh auto-detects this and exports BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1 (rewrites the URL → host.docker.internal + adds --add-host). If you bypass the wrapper and run benchlocal-cli directly against a localhost endpoint, set that env var yourself — otherwise the in-sandbox agent never reaches the model and hermes silently scores ~0/20. Failure signature: uniform ~timeout-length per-scenario latencies + flat GPU (not turn_count, which is 0 for hermes regardless of engagement). Server-side corollary (2026-07-20, #741): the URL rewrite only helps if the server accepts non-loopback connections. Engines that bind 127.0.0.1 by default (common for non-compose/BYO engines — every compose we ship binds 0.0.0.0) refuse the sandbox's connection: hermes scores 0/20 with Connection refused ... host.docker.internal in the per-scenario detail. Fix is server-side — start it with --host 0.0.0.0 (or equivalent).

  3. Verifier translation is lossy in places — the upstream BenchLocal evaluators have partial-credit branches we collapsed to pass/fail. See benchlocal-cli's docs/EXTRACTOR_NOTES.md for the specific surfaces. 4b. cli-40 variance is MODE-DEPENDENT (2026-07-20, #662). @henrykrinkle01 ran cli-40 at 4 draws per scenario in both modes on one artifact: thinking-OFF is near-deterministic — 1 flaky scenario of 40 (everything else 4/4 or 0/4) — while thinking-ON is genuinely churny (~9 flaky of 40). Consequences for reading scores: a think-off cli-40 difference between artifacts is not explainable by draw variance and should be treated as signal; a think-on cli-40 difference from single draws should not. --repeat 3 (and @seanyourhighness's 7-draw thawed-vs-flipped resolution) is worth the time on the think-on leg specifically. Thinking also redistributes rather than lifts uniformly: some scenarios go 0/4 → 4/4, others 4/4 → 1/4, and some think-on failures are token_limit (budget) rather than capability.

  4. Single-run sampling at temperature 0 — each scenario runs once, greedy, by default (see Sampling & temperature for the non-canonical override modes). For non-determinism debugging, use benchlocal-cli run --pack <id> --repeat N.

For the full pipeline architecture + JSONL pack format, read benchlocal-cli's docs.

Filing quality regressions

If quality-test.sh shows a meaningful regression (e.g., ToolCall-15 drops from 14/15 to 8/15 after an engine pin bump), file an issue with:

  1. The compose name + the change that triggered it (engine pin bump? new quant? cudagraph mode?)
  2. The full JSON output from results/quality/
  3. The pre-change baseline JSON for diff
  4. Output of bash scripts/report.sh --bench for context (vLLM image SHA, hardware)

The JSON blobs include enough per-scenario detail to reproduce specific failing scenarios via benchlocal-cli reproduce (post-v0.2 subcommand) for upstream debugging.