From b3b3282dd4f45721b0ed0809fedea1fa89e28df0 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Fri, 3 Jul 2026 10:25:00 +0200 Subject: [PATCH 1/4] docs(eval): document v4.42.4 eval migrations --- skills-data/agentv-eval-migrations/SKILL.md | 13 +- .../references/breaking-changes.md | 914 ++++++++++++++++-- 2 files changed, 838 insertions(+), 89 deletions(-) diff --git a/skills-data/agentv-eval-migrations/SKILL.md b/skills-data/agentv-eval-migrations/SKILL.md index 9b1e8f3f6..68756105a 100644 --- a/skills-data/agentv-eval-migrations/SKILL.md +++ b/skills-data/agentv-eval-migrations/SKILL.md @@ -10,8 +10,11 @@ description: >- Use this skill when updating existing AgentV eval YAML, examples, docs, or generated eval authoring guidance after a schema-breaking change. -Before editing, read `references/breaking-changes.md` and compare the eval file -against the current portable contract: +Before editing, read `references/breaking-changes.md`. For stale evals from +AgentV v4.42.4, use that reference as the migration map: it lists the +v4.42.4-era shape, current shape, migration steps, verification commands, and +compatibility notes for each major breaking authoring change. Then compare the +eval file against the current portable contract: - Keep committed eval YAML portable: prompts, cases, assertions, workspace templates, repos, hooks, env checks, Docker preflight/container config, and @@ -24,5 +27,7 @@ against the current portable contract: - Keep wire-format fields in `snake_case` and TypeScript internals in `camelCase`. -After migration, validate with `agentv eval --dry-run` when possible, or -run the repo's parser/schema tests for generated examples and fixtures. +After migration, validate with `bun apps/cli/src/cli.ts validate ` when +working in the AgentV repo, or `agentv validate ` from an installed CLI. +Run the repo's parser/schema tests for generated examples and fixtures when the +change affects shared skill data or examples. diff --git a/skills-data/agentv-eval-migrations/references/breaking-changes.md b/skills-data/agentv-eval-migrations/references/breaking-changes.md index f3eb4073e..103a8073d 100644 --- a/skills-data/agentv-eval-migrations/references/breaking-changes.md +++ b/skills-data/agentv-eval-migrations/references/breaking-changes.md @@ -1,59 +1,296 @@ -# AgentV Eval YAML Breaking Changes +# AgentV Eval YAML Breaking Changes From v4.42.4 + +This reference is for migrating eval YAML authored against AgentV v4.42.4 to +the current schema. It is intentionally migration-specific: each section states +the v4.42.4-era shape, the current shape, migration steps, verification, and +compatibility notes. + +Evidence audited for this guide: + +- v4.42.4 docs under `apps/web/src/content/docs/docs/v4.42.4/`. +- Current docs under `apps/web/src/content/docs/docs/next/`. +- v4.42.4 source from the local `v4.42.4` git tag. +- Current parser, validator, and schema code in `packages/core/src/evaluation/`. + +Use the current clone as the final source of truth. Do not claim a migration is +required only because the current docs prefer a newer convention; require it +only when the current schema, validator, or loader rejects the old shape or the +old shape no longer means the same thing. + +## Quick Migration Checklist + +For a v4.42.4-era eval: + +1. Move top-level `execution.target` to top-level `target`. +2. Move top-level `execution.targets` to top-level `targets`, and rename target + object `name` to `label`. +3. Rename suite/test/turn `assertions` to `assert`. +4. If a test uses `tests[].execution.assertions` or + `tests[].execution.evaluators`, rename that nested list to + `tests[].execution.assert`; prefer top-level or per-test `assert` for normal + grader authoring. +5. Replace `type: rubrics` or `type: rubric` with `type: llm-rubric` and move + rubric arrays from `criteria`/`rubrics` to `value`. +6. Replace `type: code-grader` and `type: code-judge` with `type: script`. +7. Move repeat/trial policy from `execution.trials` to + `evaluate_options.repeat`. +8. Move suite budget from `execution.budget_usd` to + `evaluate_options.budget_usd`. +9. Move authored suite concurrency from `execution.workers` to + `evaluate_options.max_concurrency`, or leave it to `--workers` / + project config if it is operator policy. +10. Remove top-level `execution`; current eval YAML rejects it. +11. Replace `workspace.isolation: shared|per_test` with + `workspace.scope: suite|attempt`. +12. Remove `workspace.mode` and `workspace.path` from committed eval YAML. + Use `--workspace-path` or `.agentv/config.local.yaml` for local static + directories. +13. Replace workspace hook `script:` with `command:`. +14. For executable setup, prefer top-level `extensions`; keep + `workspace.hooks.after_each.reset` for reset policy. +15. Keep raw cases under `tests` or `imports.tests`; import full eval suites + with `imports.suites`. +16. Validate with `bun apps/cli/src/cli.ts validate `. + +## Assertions Renamed To `assert` + +### v4.42.4 Shape + +v4.42.4 docs and schema used `assertions` for suite-level and per-test graders: -This reference tracks schema changes that eval authors and migration agents -should apply when modernizing AgentV eval files. +```yaml +assertions: + - name: correctness + type: llm-grader + prompt: ./graders/correctness.md + +tests: + - id: addition + criteria: Correctly calculates 15 + 27 = 42 + input: What is 15 + 27? + expected_output: "42" + assertions: + - type: contains + value: "42" +``` -## Eval Runtime Policy Moved Out Of `experiment` +v4.42.4 also accepted `execution.assertions`, `execution.evaluators`, and +case-level `evaluators` as legacy grader lists. -Do not author top-level `experiment:` in eval YAML. The whole eval file defines -the experiment; top-level `name` is the result namespace, top-level `target` -identifies the system under test, and top-level runtime/gating controls own -repeat behavior, timeout, threshold, and budget. +### Current Shape -Old: +Current eval YAML uses `assert`: ```yaml -name: backend-with-skills -experiment: - target: copilot--claude-opus-4.8 - model: claude-opus-4.8 - runs: 3 - timeout_seconds: 600 - threshold: 0.8 - budget_usd: 5 +assert: + - name: correctness + type: llm-rubric + prompt: ./graders/correctness.md + +tests: + - id: addition + input: What is 15 + 27? + expected_output: "42" + assert: + - type: contains + value: "42" + - Correctly calculates the answer +``` + +Current `EvalFileSchema` has top-level `assert` and test-level `assert`; it +does not define top-level or test-level `assertions`. Current parser code reads +case `assert` and `execution.assert`; assertion template files are expected to +contain top-level `assert`. + +### Migration Steps + +- Rename top-level `assertions:` to `assert:`. +- Rename every test-level `assertions:` to `assert:`. +- Rename conversation turn `assertions:` to `assert:`. +- Rename assertion template files from: + + ```yaml + assertions: + - type: is-json + ``` + + to: + + ```yaml + assert: + - type: is-json + ``` + +- Replace `evaluators:` with `assert:`. +- If a test had `execution.skip_defaults: true`, keep it; that flag still + controls whether suite-level defaults are appended. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "assertions:|evaluators:" path/to/evals path/to/.agentv/templates ``` -New: +### Compatibility Notes + +Result artifacts and grader stdout still contain `assertions` arrays. Do not +rename result JSON or script-grader output fields to `assert`; this migration is +for authored eval YAML and assertion template YAML. + +`criteria` remains a valid optional case field, but it is no longer the +preferred way to express the whole semantic contract. Put actual grading checks +in `assert`, usually as plain strings. + +## `criteria` Is Optional, `expected_output` Is Passive + +### v4.42.4 Shape + +v4.42.4 docs treated `criteria` as required, and when no `assertions` were +present a default `llm-grader` evaluated the case against `criteria`: ```yaml -name: backend-with-skills -target: - provider: copilot-sdk - model: claude-opus-4.8 +tests: + - id: simple-eval + criteria: Assistant correctly explains the bug and proposes a fix + input: "Debug this function..." +``` -timeout_seconds: 600 +### Current Shape + +Current docs and schema make `criteria` optional. Authored graders live under +`assert`. Plain strings in `assert` become an `llm-rubric` check. `expected_output` +is reference data available to graders; by itself it does not choose a grader. + +```yaml +tests: + - id: simple-eval + input: "Debug this function..." + expected_output: The answer explains the root cause and fix. + assert: + - Assistant correctly explains the bug + - Assistant proposes a concrete fix +``` + +### Migration Steps + +- If old `criteria` is the only grading contract, move or copy it into + `assert` as one or more plain strings. +- Keep `criteria` only when multiple graders need shared context that is not + itself the asserted checklist. +- If `expected_output` was being used as "the rubric", add explicit `assert` + entries that state how the reference should be used. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "criteria:" path/to/evals +``` + +For any remaining `criteria`, confirm it is shared grader context and not a +duplicate of `assert`. + +### Compatibility Notes + +Current runtime still carries `EvalTest.criteria` internally because prompt +templates and custom graders can consume it. The breaking authoring change is +that workers should not depend on missing `assert` plus `criteria` to define +the whole grading contract for new migrated YAML. + +## Top-Level `execution` Removed From Eval YAML + +### v4.42.4 Shape + +v4.42.4 docs used top-level `execution` for target selection, workers, error +tolerance, thresholds, budgets, trials, and suite-level graders: + +```yaml +execution: + target: azure-base + workers: 4 + fail_on_error: false + threshold: 0.8 + budget_usd: 2.00 + trials: + count: 3 + strategy: pass_at_k + +assertions: + - type: contains + value: READY +``` + +### Current Shape + +Current eval YAML rejects top-level `execution`. Move run controls to their +dedicated fields: + +```yaml +target: azure-base threshold: 0.8 evaluate_options: + max_concurrency: 4 + budget_usd: 2.00 repeat: count: 3 strategy: pass_any - early_exit: false - budget_usd: 5 + +assert: + - type: contains + value: READY ``` -## Repeat Policy Uses `evaluate_options.repeat` +### Migration Steps -Do not author top-level `runs`, top-level `early_exit`, or -top-level `repeat`. +- `execution.target` -> top-level `target`. +- `execution.targets` -> top-level `targets`. +- `execution.threshold` -> top-level `threshold`. +- `execution.budget_usd` -> `evaluate_options.budget_usd`. +- `execution.workers` -> `evaluate_options.max_concurrency` when authored + suite concurrency is part of the eval. If it is operator policy, use + `--workers` or `.agentv/config.yaml` / `agentv.config.*` `execution.workers` + instead. +- `execution.fail_on_error` has no current eval-YAML home. Treat it as + operational policy; do not commit it into migrated eval YAML. +- `execution.cache` has no current eval-YAML home. Use project/operator config + if cache policy is needed. +- Delete top-level `execution` after moving supported fields. -Old: +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "^execution:|fail_on_error:|cache:" path/to/evals +``` + +### Compatibility Notes + +Per-test `execution` still exists, but it is narrow: current schema allows +case-level `execution.assert`, `skip_defaults`, cache/fail fields, budget, and +threshold, while target selection belongs at top level or CLI. Do not keep +target selection under `tests[].execution.target` when migrating. + +## Repeat Policy Replaces `trials`, `runs`, And `early_exit` + +### v4.42.4 Shape + +v4.42.4 used `execution.trials` with strategies such as `pass_at_k`: ```yaml -runs: 3 -early_exit: true +execution: + trials: + count: 3 + strategy: pass_at_k + cost_limit_usd: 1.00 ``` -New: +Some stale evals from later intermediate branches may instead have top-level +`runs`, `repeat`, or `early_exit`. + +### Current Shape + +Current authoring uses `evaluate_options.repeat`: ```yaml evaluate_options: @@ -61,113 +298,620 @@ evaluate_options: count: 3 strategy: pass_any early_exit: true + cost_limit_usd: 1.00 +``` + +The shorthand is also accepted: + +```yaml +evaluate_options: + repeat: 3 ``` -Use `evaluate_options.repeat.strategy: pass_any` for "pass if any completed -sample passes" and `evaluate_options.repeat.strategy: pass_all` for "pass only -if every completed sample passes". `evaluate_options.repeat.early_exit` is only -a scheduling optimization; omit it or set it to `false` when you want every -sample collected for variance analysis. +### Migration Steps -## Workspace Scope Replaces Isolation +- `execution.trials.count` -> `evaluate_options.repeat.count`. +- `execution.trials.strategy: pass_at_k` -> `pass_any`. +- `execution.trials.strategy: mean` -> `mean`. +- `execution.trials.strategy: confidence_interval` -> `confidence_interval`. +- `execution.trials.cost_limit_usd` -> `evaluate_options.repeat.cost_limit_usd`. +- Top-level `runs` -> `evaluate_options.repeat.count`. +- Top-level `repeat` -> `evaluate_options.repeat`. +- Top-level `early_exit` -> `evaluate_options.repeat.early_exit`. -Old: +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "trials:|pass_at_k|^runs:|^repeat:|^early_exit:" path/to/evals +``` + +### Compatibility Notes + +Runtime types and some result artifacts still use the internal word `trials`. +Authored YAML should use `repeat`; produced executions are attempts. + +## `experiment` Is A Label, Not A Runtime Container + +### v4.42.4 Shape + +v4.42.4 public eval docs did not use a top-level `experiment` object for eval +runtime policy; runtime policy lived mostly under top-level `execution`. Some +stale evals from later development snapshots may have an object like: + +```yaml +experiment: + target: codex + model: gpt-5 + runs: 3 + timeout_seconds: 600 +``` + +### Current Shape + +Current schema accepts top-level `experiment` only as a non-empty string run +grouping label. Current docs also support promptfoo-shaped `tags.experiment`. + +```yaml +experiment: with-skills +target: codex +timeout_seconds: 600 +evaluate_options: + repeat: + count: 3 + strategy: pass_any +``` + +or: + +```yaml +tags: + experiment: with-skills +target: codex +``` + +### Migration Steps + +- If `experiment` is an object, move runtime fields out: + - target identity -> top-level `target` or `targets` + - repeat/runs -> `evaluate_options.repeat` + - budget -> `evaluate_options.budget_usd` + - timeout -> top-level `timeout_seconds` + - threshold -> top-level `threshold` +- Keep only a string label in `experiment`, or use `tags.experiment`. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "^experiment:" path/to/evals +``` + +### Compatibility Notes + +Do not describe a v4.42.4 eval as if `experiment:` was already the main runtime +object unless you verified that exact file or commit. For v4.42.4 migrations, +the common source shape is `execution:`. + +## Workspace Lifetime: `isolation` To `scope` + +### v4.42.4 Shape + +v4.42.4 workspace docs used `isolation`: ```yaml workspace: - isolation: per_case + repos: + - path: ./repo + repo: org/repo + commit: main + hooks: + after_each: + reset: fast + isolation: shared # shared | per_test ``` -New: +### Current Shape + +Current workspace docs and schema use `scope`: ```yaml workspace: - scope: attempt + repos: + - path: ./repo + repo: org/repo + commit: main + hooks: + after_each: + reset: fast + scope: suite # suite | attempt ``` -Use `scope: attempt` for a clean workspace per resolved execution attempt. Use -`scope: suite` when cases intentionally share one prepared workspace. +### Migration Steps + +- `workspace.isolation: shared` -> `workspace.scope: suite`. +- `workspace.isolation: per_test` -> `workspace.scope: attempt`. +- If `isolation` is omitted, preserve behavior by omitting `scope` or setting + `scope: suite`. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "isolation:" path/to/evals +``` -## Suite Wrapper Workspace Ownership +### Compatibility Notes -Eval files that import child eval suites with `type: suite` cannot define a -parent `workspace`. Imported suites own their task environment, including repos, -templates, hooks, Docker config, env checks, and isolation. +`scope: attempt` means a clean workspace for each resolved execution attempt: +prompt/target/test/repeat expansion. Docker configuration does not replace +workspace lifetime; use `workspace.scope` even when `workspace.docker` exists. -If the parent should own workspace context, import raw cases with `type: tests` -or direct path shorthand instead of importing suites. +## Local Workspace Modes Removed From Eval YAML -## Runtime Workspace Blocks Removed From Eval YAML +### v4.42.4 Shape -Do not author these blocks in eval YAML: +v4.42.4 workspace docs allowed `mode` and `path`: ```yaml -experiment: - workspace: - mode: static - path: /path/to/local/workspace +workspace: + mode: static # pooled | temp | static + path: /tmp/my-ws +``` -execution: - workspace: - mode: static - path: /path/to/local/workspace +### Current Shape + +Current committed eval YAML must keep portable workspace setup under +`workspace` and put machine-local existing directories outside the eval: + +```yaml +workspace: + repos: + - path: ./repo + repo: org/repo + commit: main + scope: suite ``` -Existing local workspace directories are machine-local runtime bindings. Use one -of these instead: +One-off local binding: ```bash -agentv eval evals/my-eval.yaml --workspace-path /path/to/local/workspace +bun apps/cli/src/cli.ts eval path/to/eval.eval.yaml --workspace-path /tmp/my-ws ``` +Persistent local binding: + ```yaml # .agentv/config.local.yaml execution: - workspace_path: /path/to/local/workspace + workspace_path: /tmp/my-ws +``` + +### Migration Steps + +- Remove `workspace.mode`. +- Remove `workspace.path`, `workspace.static_path`, `workspace.static`, and + `workspace.pool` if present. +- Convert portable setup to `workspace.template`, `workspace.repos`, + `workspace.hooks.after_each.reset`, `workspace.env`, and `workspace.scope`. +- Put existing local paths in `--workspace-path` or `.agentv/config.local.yaml`. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "workspace:|mode:|path:|static_path:|pool:" path/to/evals ``` -Keep portable task setup under top-level or case-level `workspace`. +Inspect matches manually because `repos[].path` is still valid. + +### Compatibility Notes + +Current project config intentionally strips `execution.workspace_path` from +committed `.agentv/config.yaml`; it is only supported in `config.local.yaml`. + +## Workspace Hooks And Lifecycle Extensions + +### v4.42.4 Shape -## Workspace Mode and Path Removed From Eval YAML +v4.42.4 workspace hooks allowed `command` and `script` fields, and docs used +workspace hooks for executable setup: -Do not author `workspace.mode`, `workspace.path`, `workspace.static_path`, -`workspace.static`, or `workspace.pool` in eval YAML. +```yaml +workspace: + hooks: + before_all: + script: ["bun", "run", "setup.ts"] + after_each: + command: ["bun", "run", "reset.ts"] + reset: fast +``` -Old: +### Current Shape + +Current hook configs reject `script`; use `command`. Current docs prefer +top-level `extensions` for executable setup and keep +`workspace.hooks.after_each.reset` for reset policy: ```yaml +extensions: + - file://scripts/setup.mjs:beforeAll + workspace: - mode: static - path: /path/to/local/workspace + hooks: + after_each: + reset: fast ``` -New portable eval YAML: +Legacy command hooks still parse for existing suites when they use `command`: + +```yaml +workspace: + hooks: + before_all: + command: ["bun", "run", "setup.ts"] +``` + +### Migration Steps + +- Rename any hook `script:` field to `command:`. +- Prefer moving setup/fixture/build/install commands to top-level + `extensions`. +- Keep `workspace.hooks.after_each.reset` for `none`, `fast`, or `strict`. +- Extension file references must use `file://path/to/hook.ts:beforeAll` style + and a Promptfoo-compatible hook name: `beforeAll`, `beforeEach`, + `afterEach`, or `afterAll`. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "script:|extensions:" path/to/evals +``` + +### Compatibility Notes + +`workspace.hooks` and `target.hooks` use snake_case lifecycle keys such as +`before_all`; top-level `extensions` use Promptfoo-compatible camel-case hook +suffixes in `file://...:beforeAll`. + +## Repository Entries Are Provenance Only + +### v4.42.4 Shape + +v4.42.4 already documented `workspace.repos[].repo`, `commit`, +`base_commit`, `ancestor`, and `sparse` as repo provenance. The v4.42.4 parser +also rejected some acquisition fields such as `source`, `checkout`, and +`clone`. + +### Current Shape + +Current parser continues to keep acquisition out of eval YAML and additionally +rejects `type`, `resolve`, and `resolver`: ```yaml workspace: repos: - path: ./repo repo: org/repo - commit: main - scope: suite + base_commit: abc123def + ancestor: 1 + sparse: + - packages/core ``` -Optional local binding: +### Migration Steps + +- `workspace.repos[].source` -> `workspace.repos[].repo`. +- `workspace.repos[].checkout.ref` or similar -> `commit` / `base_commit`. +- `workspace.repos[].clone.sparse` -> top-level `sparse`. +- Remove `type`, `resolve`, and `resolver` from repo entries. +- Configure acquisition policy in repo resolver/project config, not in eval + YAML. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "source:|checkout:|clone:|type:|resolve:|resolver:" path/to/evals +``` + +Inspect `type:` matches manually because grader entries still use `type`. + +### Compatibility Notes + +`repos[].path` remains valid and means the target directory inside the +materialized workspace. It is not the removed local `workspace.path`. + +## Target And Runtime Separation + +### v4.42.4 Shape + +v4.42.4 docs selected targets under `execution` and used `name` in target +objects: ```yaml -# .agentv/config.local.yaml execution: - workspace_path: /path/to/local/workspace + target: azure-base + targets: + - baseline + - name: with-skills + use_target: default + hooks: + before_each: + command: ["setup-plugins.sh", "skills"] +``` + +v4.42.4 `.agentv/targets.yaml` examples also used `name` and provider settings +as top-level target fields: + +```yaml +targets: + - name: azure-base + provider: azure + endpoint: ${{ AZURE_OPENAI_ENDPOINT }} + api_key: ${{ AZURE_OPENAI_API_KEY }} + model: ${{ AZURE_DEPLOYMENT_NAME }} +``` + +### Current Shape + +Current eval YAML uses top-level `target` or `targets`. Target references use +`label`; `id` is reserved for provider/backend identity when needed: + +```yaml +target: azure-base + +targets: + - label: with-skills + use_target: default + hooks: + before_each: + command: ["setup-plugins.sh", "skills"] ``` -Harness-managed repo workspaces use temp materialization by default. Use -`workspace.scope: suite | attempt` for portable lifetime. Use `--workspace-path` -or `execution.workspace_path` when an existing directory should be used as-is. +Current `.agentv/targets.yaml` uses `label` and nests provider settings under +`config`: + +```yaml +targets: + - label: azure-base + provider: azure + config: + endpoint: ${{ AZURE_OPENAI_ENDPOINT }} + api_key: ${{ AZURE_OPENAI_API_KEY }} + model: ${{ AZURE_DEPLOYMENT_NAME }} +``` -## Docker Is Not Folder Isolation +### Migration Steps -`workspace.docker` describes environment, preflight, or container bindings. It -does not replace `workspace.scope`. +- Eval YAML: `execution.target` -> top-level `target`. +- Eval YAML: `execution.targets` -> top-level `targets`. +- Eval target object `name` -> `label`. +- If an eval-local target object has provider configuration, include a + `label`; use `extends` to derive from a base target. +- Targets file: rename `name` to `label` and move provider-specific settings + into `config`. +- Keep AgentV target extensions such as `grader_target`, `use_target`, + `fallback_targets`, `workers`, and `batch_requests` as top-level fields on + target objects. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +bun apps/cli/src/cli.ts validate .agentv/targets.yaml +rg -n "execution:|name:|endpoint:|api_key:|model:" path/to/evals .agentv/targets.yaml +``` + +Inspect `name:` matches manually because grader names still exist. + +### Compatibility Notes + +Do not put target selection in test cases when migrating. Split +target-specific cases into separate eval suites, use tags/filters, or run the +same eval with different `--target` values. + +## Grader Type And Rubric Shape Changes + +### v4.42.4 Shape + +v4.42.4 docs used: + +```yaml +assertions: + - type: rubrics + criteria: + - id: accuracy + outcome: Correctly identifies the denied party + weight: 5 + + - type: code-grader + command: ["python", "check.py"] +``` + +v4.42.4 accepted snake_case grader types such as `is_json`. + +### Current Shape + +Current authored grader types are kebab-case. Semantic rubric grading uses +plain assertion strings or `llm-rubric`: + +```yaml +assert: + - type: llm-rubric + value: + - id: accuracy + outcome: Correctly identifies the denied party + weight: 5 + + - type: script + command: ["python", "check.py"] +``` + +### Migration Steps + +- `type: rubrics` or `type: rubric` -> `type: llm-rubric`. +- `criteria:` / `rubrics:` / `rubric_item:` under `llm-rubric` -> + `value:`. +- `type: g-eval` -> `type: llm-rubric`. +- `type: code-grader`, `code-judge`, `code_grader`, or `code_judge` -> + `type: script`. +- `type: llm_judge` or `llm_grader` -> `type: llm-grader`. +- Convert multi-word snake_case deterministic types to kebab-case: + `is_json` -> `is-json`, `contains_all` -> `contains-all`, + `starts_with` -> `starts-with`, and so on. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "type: (rubrics|rubric|g-eval|code-grader|code-judge|code_grader|code_judge|llm_judge|llm_grader|.*_.*)" path/to/evals +``` + +### Compatibility Notes + +Current `grader-parser.ts` contains replacement hints for several removed type +names, but current schema and generated references are stricter than v4.42.4. +Migrate authored YAML instead of relying on parser leniency. + +Script grader output still returns JSON with an `assertions` array; do not +rename grader output to `assert`. + +## Prompt File Paths And `file://` + +### v4.42.4 Shape + +v4.42.4 LLM grader docs allowed both: + +```yaml +assertions: + - type: llm-grader + prompt: ./graders/correctness.md + - type: llm-grader + prompt: file://graders/correctness.md +``` + +### Current Shape + +Current LLM grader docs keep the same prompt path behavior: + +```yaml +assert: + - type: llm-rubric + prompt: ./graders/correctness.md + - type: llm-rubric + prompt: file://graders/correctness.md +``` + +Other current file-reference surfaces are stricter: + +```yaml +extensions: + - file://scripts/setup.mjs:beforeAll + +default_test: file://defaults.yaml + +tests: + - file://cases.yaml +``` + +### Migration Steps + +- Do not add `file://` to ordinary `prompt: ./path.md` only for migration; the + current prompt resolver still treats path-like strings as file references. +- Keep `file://` when you need explicit file-reference resolution. +- Add `file://` for top-level `extensions` entries. +- Use `file://` or `ref://` for `default_test` string references. +- In `tests` arrays, use `file://...` for file include entries. A top-level + `tests: ./cases.yaml` string remains accepted for raw case files. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "prompt:|extensions:|default_test:|file://" path/to/evals +``` + +### Compatibility Notes + +This is partly a non-breaking area: v4.42.4 and current docs both allow +relative prompt paths. The breaking part is the newer surfaces that require +`file://`, especially `extensions`. + +## Imports, Raw Cases, And Suite Ownership + +### v4.42.4 Shape + +v4.42.4 eval files used `tests` for inline cases or external raw case files: + +```yaml +name: my-eval +execution: + target: default +tests: ./cases.yaml +``` + +The external file contained raw case rows. + +### Current Shape + +Current eval YAML still accepts inline `tests` and `tests: ./cases.yaml`, but +adds explicit imports for composition: + +```yaml +imports: + suites: + - path: ../suites/refunds.eval.yaml + tests: + - path: ../cases/refund-smoke.cases.yaml + +tests: + - id: local-edge-case + input: Can a final-sale item be refunded after damage in transit? + assert: + - Explains the final-sale exception +``` + +### Migration Steps + +- Keep `tests: ./cases.yaml` when the file is a raw case array, JSONL, CSV, + directory, glob, or script-backed dataset. +- Use `imports.tests` when importing raw rows into the parent suite context. +- Use `imports.suites` when importing full child eval suites that own their + own `workspace`, input, assertions, and task environment. +- Do not define a parent `workspace` in a wrapper eval that imports child + suites through `imports.suites`; child suites own their environments. +- Replace legacy `tests[].include` entries with `imports.suites` or + `imports.tests` where possible. +- Use `run:` on import entries only for scoped overrides: + `threshold`, `repeat`, `timeout_seconds`, and `budget_usd`. + +### Verification + +```bash +bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml +rg -n "include:|imports:|tests:" path/to/evals +``` + +### Compatibility Notes + +`eval_cases` remains a deprecated alias in the current schema, but migrated +YAML should use `tests`. The current convention is that runnable suites use +`*.eval.yaml`; reusable raw case files commonly use `*.cases.yaml` or JSONL. + +## Result Artifact Path Changes Are Not Eval YAML Migrations + +v4.42.4 docs described local run workspaces under +`.agentv/results/runs///`. Current docs describe v2 run +workspaces under `.agentv/results//`, with experiment metadata stored +in `summary.json` / rows rather than inferred from the path. + +Do not edit eval YAML just to chase result artifact path changes. Migrate only +authored fields that the eval parser reads. Use: + +```bash +bun apps/cli/src/cli.ts results validate path/to/run-dir +``` -Use `workspace.scope: suite | attempt` for workspace lifetime, regardless of -whether Docker is configured. +for existing run artifacts. From f325e2c0cec7c5e7cd9c17edecba9fc899f0fb99 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Fri, 3 Jul 2026 10:33:44 +0200 Subject: [PATCH 2/4] docs(eval): prefer commit in workspace repo migrations --- .../references/breaking-changes.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/skills-data/agentv-eval-migrations/references/breaking-changes.md b/skills-data/agentv-eval-migrations/references/breaking-changes.md index 103a8073d..fa0317849 100644 --- a/skills-data/agentv-eval-migrations/references/breaking-changes.md +++ b/skills-data/agentv-eval-migrations/references/breaking-changes.md @@ -579,10 +579,10 @@ suffixes in `file://...:beforeAll`. ### v4.42.4 Shape -v4.42.4 already documented `workspace.repos[].repo`, `commit`, -`base_commit`, `ancestor`, and `sparse` as repo provenance. The v4.42.4 parser -also rejected some acquisition fields such as `source`, `checkout`, and -`clone`. +v4.42.4 already documented `workspace.repos[].repo`, `commit`, `ancestor`, and +`sparse` as repo provenance. It also accepted `base_commit` as a +SWE-bench-friendly alias for `commit`. The v4.42.4 parser rejected some +acquisition fields such as `source`, `checkout`, and `clone`. ### Current Shape @@ -594,7 +594,7 @@ workspace: repos: - path: ./repo repo: org/repo - base_commit: abc123def + commit: abc123def ancestor: 1 sparse: - packages/core @@ -603,7 +603,7 @@ workspace: ### Migration Steps - `workspace.repos[].source` -> `workspace.repos[].repo`. -- `workspace.repos[].checkout.ref` or similar -> `commit` / `base_commit`. +- `workspace.repos[].checkout.ref` or similar -> `commit`. - `workspace.repos[].clone.sparse` -> top-level `sparse`. - Remove `type`, `resolve`, and `resolver` from repo entries. - Configure acquisition policy in repo resolver/project config, not in eval @@ -622,6 +622,9 @@ Inspect `type:` matches manually because grader entries still use `type`. `repos[].path` remains valid and means the target directory inside the materialized workspace. It is not the removed local `workspace.path`. +`base_commit` remains accepted as an alias for `commit`, mainly for +SWE-bench-style datasets, but migrated examples should use `commit` unless the +source dataset convention specifically uses `base_commit`. ## Target And Runtime Separation From 973b8fa0b64da52c138e29536902a9e468bd87ae Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Fri, 3 Jul 2026 11:00:00 +0200 Subject: [PATCH 3/4] fix(eval): remove base_commit repo alias --- apps/cli/src/commands/prepare/index.ts | 4 ---- .../docs/docs/next/evaluation/eval-cases.mdx | 4 ++-- .../docs/docs/next/evaluation/eval-files.mdx | 5 ++-- .../next/guides/workspace-architecture.mdx | 19 +++++++-------- .../docs/docs/next/targets/configuration.mdx | 7 +++--- .../content/docs/docs/next/tools/import.mdx | 4 ++-- examples/features/docker-workspace/README.md | 4 ++-- examples/showcase/bug-fix-benchmark/README.md | 4 ++-- .../evals/bug-fixes.eval.yaml | 2 +- .../core/src/evaluation/prepared-workspace.ts | 2 -- packages/core/src/evaluation/types.ts | 4 +--- .../evaluation/validation/eval-file.schema.ts | 6 +---- .../evaluation/validation/eval-validator.ts | 24 +++++++++---------- .../src/evaluation/workspace/repo-checkout.ts | 4 ++-- .../workspace/repo-config-parser.ts | 21 +++++++--------- .../evaluation/repo-schema-validation.test.ts | 7 +++--- .../validation/eval-validator.test.ts | 9 +++---- .../workspace-config-parsing.test.ts | 16 ++++++------- .../evaluation/workspace/deps-scanner.test.ts | 7 +++--- .../evaluation/workspace/repo-manager.test.ts | 4 ++-- .../workspace/script-executor.test.ts | 2 +- packages/sdk/src/eval.ts | 2 -- scripts/import-huggingface.py | 4 ++-- .../agentv-bench/references/eval-yaml-spec.md | 6 ++--- .../references/breaking-changes.md | 18 +++++++------- skills-data/agentv-eval-review/SKILL.md | 2 +- skills-data/agentv-eval-writer/SKILL.md | 15 +++++------- .../references/eval.schema.json | 16 ------------- 28 files changed, 91 insertions(+), 131 deletions(-) diff --git a/apps/cli/src/commands/prepare/index.ts b/apps/cli/src/commands/prepare/index.ts index 84a8eb426..8ed3a57a4 100644 --- a/apps/cli/src/commands/prepare/index.ts +++ b/apps/cli/src/commands/prepare/index.ts @@ -37,7 +37,6 @@ interface RepoPin { readonly path?: string; readonly repo?: string; readonly commit?: string; - readonly baseCommit?: string; readonly ancestor?: number; readonly sparse?: readonly string[]; } @@ -89,7 +88,6 @@ interface RepoPinWire { readonly path?: string; readonly repo?: string; readonly commit?: string; - readonly base_commit?: string; readonly ancestor?: number; readonly sparse?: readonly string[]; } @@ -129,7 +127,6 @@ function toRepoPins(pins: readonly PreparedWorkspaceRepoPin[]): readonly RepoPin ...(pin.path !== undefined && { path: pin.path }), ...(pin.repo !== undefined && { repo: pin.repo }), ...(pin.commit !== undefined && { commit: pin.commit }), - ...(pin.baseCommit !== undefined && { baseCommit: pin.baseCommit }), ...(pin.ancestor !== undefined && { ancestor: pin.ancestor }), ...(pin.sparse !== undefined && { sparse: pin.sparse }), })); @@ -242,7 +239,6 @@ function toManifestWire(result: PrepareResult): PrepareManifestWire { ...(pin.path !== undefined && { path: pin.path }), ...(pin.repo !== undefined && { repo: pin.repo }), ...(pin.commit !== undefined && { commit: pin.commit }), - ...(pin.baseCommit !== undefined && { base_commit: pin.baseCommit }), ...(pin.ancestor !== undefined && { ancestor: pin.ancestor }), ...(pin.sparse !== undefined && { sparse: pin.sparse }), })), diff --git a/apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx b/apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx index 787ee212c..485e1e2be 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx @@ -189,14 +189,14 @@ tests: repos: - path: ./repo repo: sympy/sympy - base_commit: "abc123def" + commit: "abc123def" hooks: before_each: command: ["python", "apply_test_patch.py"] ``` The `metadata` field is included in the stdin JSON passed to lifecycle commands as `case_metadata`. -Operational checkout state belongs under `workspace.repos[].base_commit`; matching metadata fields such as `source_commit` are informational only. +Operational checkout state belongs under `workspace.repos[].commit`; matching metadata fields such as `source_commit` are informational only. For historical repo-state evals, pin the checkout under `workspace.repos[]` instead of only mentioning the SHA in prompt prose: diff --git a/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx b/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx index 087f0ee41..fa970d56b 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx @@ -138,9 +138,8 @@ checks, scope, and repo provenance in `workspace`. Put lifecycle setup that does not acquire repos in `extensions`. For historical or repo-state evals, put the checkout under -`workspace.repos[].commit` or `workspace.repos[].base_commit`. A commit SHA in -the prompt or metadata is useful context, but it does not materialize a repo for -the agent to inspect. +`workspace.repos[].commit`. A commit SHA in the prompt or metadata is useful +context, but it does not materialize a repo for the agent to inspect. ### Prompts, Vars, and Target Expansion diff --git a/apps/web/src/content/docs/docs/next/guides/workspace-architecture.mdx b/apps/web/src/content/docs/docs/next/guides/workspace-architecture.mdx index 4990c6be5..d954334c9 100644 --- a/apps/web/src/content/docs/docs/next/guides/workspace-architecture.mdx +++ b/apps/web/src/content/docs/docs/next/guides/workspace-architecture.mdx @@ -38,7 +38,7 @@ eval start | 3. Repo materialization | For each workspace.repos entry: | a. resolve acquisition | - registered project, configured mirror, | b. git clone/fetch | AgentV cache, or remote fallback -| c. git checkout | - check out commit/base_commit/HEAD +| c. git checkout | - check out commit or HEAD +---------------------------+ | v @@ -85,19 +85,16 @@ Supported repo fields: | `path` | Directory inside the workspace where the repo is materialized | | `repo` | Repository identity: full clone URL or GitHub `org/name` shorthand | | `commit` | Branch, tag, or SHA to check out after clone | -| `base_commit` | Alias for `commit`, useful for SWE-bench-style datasets | | `sparse` | Optional sparse-checkout paths | -| `ancestor` | Walk N parents back after resolving `commit` / `base_commit` | -| `resolver` | Optional `repo_resolvers[].name` override from AgentV config | +| `ancestor` | Walk N parents back after resolving `commit` | -`commit` is the canonical AgentV checkout pin. `base_commit` exists only as a -SWE-Bench-friendly alias for the same value; when both fields are present they -must match. Prefer `commit` in new AgentV-authored evals unless preserving an -upstream dataset column name makes the eval easier to audit. +`commit` is the AgentV checkout pin. Translate upstream dataset columns such as +SWE-bench `base_commit` to `commit` when importing eval YAML. -`source`, `type`, `checkout`, `checkout.resolve`, and `clone` are not part of -the repo schema. Acquisition settings are deliberately outside eval YAML so the -same benchmark can run against the same repository identity on every machine +`source`, `type`, `checkout`, `checkout.resolve`, `clone`, `resolve`, and +`resolver` are not part of the repo schema. Acquisition settings are +deliberately outside eval YAML so the same benchmark can run against the same +repository identity on every machine while each harness uses the fastest safe local source available. ## Native workspace boundary diff --git a/apps/web/src/content/docs/docs/next/targets/configuration.mdx b/apps/web/src/content/docs/docs/next/targets/configuration.mdx index 279b3d86a..ed974c498 100644 --- a/apps/web/src/content/docs/docs/next/targets/configuration.mdx +++ b/apps/web/src/content/docs/docs/next/targets/configuration.mdx @@ -160,7 +160,7 @@ case context: "test_id": "case-01", "eval_run_id": "run-123", "case_input": "Fix the bug", - "case_metadata": { "repo": "sympy/sympy", "base_commit": "abc123" } + "case_metadata": { "repo": "sympy/sympy", "source_commit": "abc123" } } ``` @@ -192,7 +192,6 @@ workspace: | `repos[].path` | Directory within the workspace to clone into | | `repos[].repo` | Repository identity: full clone URL or GitHub `org/name` shorthand | | `repos[].commit` | Branch, tag, or SHA to check out (default: `HEAD`) | -| `repos[].base_commit` | Alias for `commit`, useful for SWE-bench-style datasets | | `repos[].ancestor` | Walk N commits back from the checked-out ref (e.g., `1` for parent) | | `repos[].sparse` | Sparse checkout paths | | `hooks.after_each.reset` | Reset policy after each test: `none`, `fast`, `strict` | @@ -227,12 +226,12 @@ workspace: after_each: reset: fast -# GitHub shorthand with a base_commit alias +# GitHub shorthand with a pinned commit workspace: repos: - path: ./repo repo: org/repo - base_commit: abc123def + commit: abc123def ``` ### Cleanup Behavior diff --git a/apps/web/src/content/docs/docs/next/tools/import.mdx b/apps/web/src/content/docs/docs/next/tools/import.mdx index af8449168..cd5a6acef 100644 --- a/apps/web/src/content/docs/docs/next/tools/import.mdx +++ b/apps/web/src/content/docs/docs/next/tools/import.mdx @@ -218,7 +218,7 @@ uv run scripts/import-huggingface.py \ Each instance becomes an EVAL.yaml with: - `input` — the problem statement - `workspace.docker.image` — the pre-built SWE-bench Docker image (`ghcr.io/epoch-research/swe-bench.eval.x86_64.:latest`) -- `workspace.repos[].base_commit` — the commit to reset to before the agent runs +- `workspace.repos[].commit` — the SWE-bench `base_commit` value to reset to before the agent runs - `assertions` — `script` tasks that run `FAIL_TO_PASS` and `PASS_TO_PASS` pytest suites inside the container Run an imported SWE-bench eval against any coding agent target: @@ -234,4 +234,4 @@ uv run scripts/import-huggingface.py \ agentv eval /tmp/swebench-eval/*.EVAL.yaml --target codex ``` -The Docker workspace spins up the pre-built SWE-bench image, checks out `base_commit`, runs the agent to apply a patch, then grades by running the test suite inside the container. +The Docker workspace spins up the pre-built SWE-bench image, checks out the imported `commit`, runs the agent to apply a patch, then grades by running the test suite inside the container. diff --git a/examples/features/docker-workspace/README.md b/examples/features/docker-workspace/README.md index 7191ca8b4..c2be56972 100644 --- a/examples/features/docker-workspace/README.md +++ b/examples/features/docker-workspace/README.md @@ -37,7 +37,7 @@ workspace: cpus: 2 # optional Docker CPU limit ``` -For evals that need a repo pinned to a dataset snapshot, use `workspace.repos[].base_commit`: +For evals that need a repo pinned to a dataset snapshot, use `workspace.repos[].commit`: ```yaml workspace: @@ -45,7 +45,7 @@ workspace: image: swebench/sweb.eval.x86_64.django__django-15180 repos: - path: /testbed - base_commit: abc123def + commit: abc123def ``` Repos defined without `repo` are assumed to already exist inside the container (e.g., SWE-bench prebuilt images). diff --git a/examples/showcase/bug-fix-benchmark/README.md b/examples/showcase/bug-fix-benchmark/README.md index 14a666a56..99f45d9f3 100644 --- a/examples/showcase/bug-fix-benchmark/README.md +++ b/examples/showcase/bug-fix-benchmark/README.md @@ -21,7 +21,7 @@ Compares four configurations on identical bug fix tasks: ``` ┌─────────────────────────────────────────────────────────────────┐ -│ 1. Clone public repo at base_commit (broken state) │ +│ 1. Clone public repo at commit (broken state) │ │ 2. Run target before_each hook (install plugin config) │ │ 3. Agent receives issue description │ │ 4. Agent diagnoses and writes fix │ @@ -81,7 +81,7 @@ The `setup-variant.sh` hook copies these files into the workspace before each te ## Adding New Test Cases 1. Find a bug fix from GitHub issues/PRs -2. Note the `base_commit` (before the fix) +2. Note the `commit` (before the fix) 3. Copy the issue description as the test `input` 4. Add assertions to verify the fix 5. Add to `evals/bug-fixes.eval.yaml` diff --git a/examples/showcase/bug-fix-benchmark/evals/bug-fixes.eval.yaml b/examples/showcase/bug-fix-benchmark/evals/bug-fixes.eval.yaml index 565e81da4..1b62d57f2 100644 --- a/examples/showcase/bug-fix-benchmark/evals/bug-fixes.eval.yaml +++ b/examples/showcase/bug-fix-benchmark/evals/bug-fixes.eval.yaml @@ -17,7 +17,7 @@ workspace: repos: - path: ./repo repo: https://github.com/EntityProcess/agentv - base_commit: "6e446b722627e9df017b22e391fa63320362d8c7" + commit: "6e446b722627e9df017b22e391fa63320362d8c7" hooks: before_each: reset: fast diff --git a/packages/core/src/evaluation/prepared-workspace.ts b/packages/core/src/evaluation/prepared-workspace.ts index d9ed761ca..786514c9d 100644 --- a/packages/core/src/evaluation/prepared-workspace.ts +++ b/packages/core/src/evaluation/prepared-workspace.ts @@ -56,7 +56,6 @@ export interface PreparedWorkspaceRepoPin { readonly path?: string; readonly repo?: string; readonly commit?: string; - readonly baseCommit?: string; readonly ancestor?: number; readonly sparse?: readonly string[]; } @@ -140,7 +139,6 @@ function toRepoPins(repos: readonly RepoConfig[] | undefined): readonly Prepared ...(repo.path !== undefined && { path: repo.path }), ...(repo.repo !== undefined && { repo: repo.repo }), ...(repo.commit !== undefined && { commit: repo.commit }), - ...(repo.base_commit !== undefined && { baseCommit: repo.base_commit }), ...(repo.ancestor !== undefined && { ancestor: repo.ancestor }), ...(repo.sparse !== undefined && { sparse: repo.sparse }), })); diff --git a/packages/core/src/evaluation/types.ts b/packages/core/src/evaluation/types.ts index 1adff67ca..3aa0de8d4 100644 --- a/packages/core/src/evaluation/types.ts +++ b/packages/core/src/evaluation/types.ts @@ -244,9 +244,7 @@ export type RepoConfig = { readonly repo?: string; /** Commit, branch, or tag to check out after materialization. */ readonly commit?: string; - /** SWE-bench-friendly alias for commit when pinning a dataset snapshot commit. */ - readonly base_commit?: string; - /** Walk this many ancestors back after checking out commit/base_commit. */ + /** Walk this many ancestors back after checking out commit. */ readonly ancestor?: number; /** Optional sparse-checkout paths. */ readonly sparse?: readonly string[]; diff --git a/packages/core/src/evaluation/validation/eval-file.schema.ts b/packages/core/src/evaluation/validation/eval-file.schema.ts index 3d59a46fd..862d52b01 100644 --- a/packages/core/src/evaluation/validation/eval-file.schema.ts +++ b/packages/core/src/evaluation/validation/eval-file.schema.ts @@ -386,14 +386,10 @@ const RepoSchema = z path: z.string().optional(), repo: z.string().min(1).optional(), commit: z.string().min(1).optional(), - base_commit: z.string().min(1).optional(), ancestor: z.number().int().min(0).optional(), sparse: z.array(z.string()).optional(), }) - .strict() - .refine((repo) => !repo.commit || !repo.base_commit || repo.commit === repo.base_commit, { - message: 'commit and base_commit must match when both are set', - }); + .strict(); const WorkspaceHookSchema = z .object({ diff --git a/packages/core/src/evaluation/validation/eval-validator.ts b/packages/core/src/evaluation/validation/eval-validator.ts index 9503ef615..f293fdd84 100644 --- a/packages/core/src/evaluation/validation/eval-validator.ts +++ b/packages/core/src/evaluation/validation/eval-validator.ts @@ -1633,7 +1633,17 @@ function validateWorkspaceRepoConfig( filePath, location: `${location}.repos[path=${repo.path ?? '(none)'}]`, message: - 'workspace.repos[].checkout has been removed. Use top-level commit, base_commit, and ancestor.', + 'workspace.repos[].checkout has been removed. Use top-level commit and ancestor.', + }); + } + + if ('base_commit' in repo) { + errors.push({ + severity: 'error', + filePath, + location: `${location}.repos[path=${repo.path ?? '(none)'}]`, + message: + 'workspace.repos[].base_commit has been removed. Use workspace.repos[].commit.', }); } @@ -1685,18 +1695,6 @@ function validateWorkspaceRepoConfig( }); } - if ( - typeof repo.commit === 'string' && - typeof repo.base_commit === 'string' && - repo.commit !== repo.base_commit - ) { - errors.push({ - severity: 'error', - filePath, - location: `${location}.repos[path=${repo.path ?? '(none)'}]`, - message: 'repos[].commit and repos[].base_commit must match when both are set.', - }); - } } } diff --git a/packages/core/src/evaluation/workspace/repo-checkout.ts b/packages/core/src/evaluation/workspace/repo-checkout.ts index a7a602061..983f8da2d 100644 --- a/packages/core/src/evaluation/workspace/repo-checkout.ts +++ b/packages/core/src/evaluation/workspace/repo-checkout.ts @@ -6,7 +6,7 @@ export interface RepoCheckoutTarget { } export function getRepoCheckoutRef(repo: RepoConfig | undefined): string { - return repo?.commit ?? repo?.base_commit ?? 'HEAD'; + return repo?.commit ?? 'HEAD'; } export function getRepoCheckoutTargets( @@ -14,7 +14,7 @@ export function getRepoCheckoutTargets( ): RepoCheckoutTarget[] { if (!repos) return []; return repos - .filter((repo) => repo.commit || repo.base_commit) + .filter((repo) => repo.commit) .map((repo) => ({ path: repo.path, ref: getRepoCheckoutRef(repo), diff --git a/packages/core/src/evaluation/workspace/repo-config-parser.ts b/packages/core/src/evaluation/workspace/repo-config-parser.ts index d0a3203b5..7a88051cc 100644 --- a/packages/core/src/evaluation/workspace/repo-config-parser.ts +++ b/packages/core/src/evaluation/workspace/repo-config-parser.ts @@ -1,10 +1,10 @@ /** * Shared parser for eval workspace repo entries. * - * Repo entries are provenance-only: `repo` names the canonical repository, - * `commit` pins the checkout, and `base_commit` is a SWE-bench-friendly alias - * for that pin. Acquisition details such as local mirrors, clone depth, filters, - * and source type are resolved by the workspace harness, not the eval YAML. + * Repo entries are provenance-only: `repo` names the canonical repository, and + * `commit` pins the checkout. Acquisition details such as local mirrors, clone + * depth, filters, and source type are resolved by the workspace harness, not + * the eval YAML. */ import type { RepoConfig } from '../types.js'; import { isJsonObject } from '../types.js'; @@ -30,9 +30,12 @@ export function parseRepoConfig(raw: unknown): RepoConfig | undefined { } if ('checkout' in obj) { throw new Error( - 'workspace.repos[].checkout has been removed. Use top-level commit, base_commit, and ancestor.', + 'workspace.repos[].checkout has been removed. Use top-level commit and ancestor.', ); } + if ('base_commit' in obj) { + throw new Error('workspace.repos[].base_commit has been removed. Use workspace.repos[].commit.'); + } if ('clone' in obj) { throw new Error('workspace.repos[].clone has been removed. Use top-level sparse if needed.'); } @@ -53,15 +56,10 @@ export function parseRepoConfig(raw: unknown): RepoConfig | undefined { const repoPath = readString(obj, 'path'); const repo = readString(obj, 'repo'); const commit = readString(obj, 'commit'); - const baseCommit = readString(obj, 'base_commit'); const ancestor = typeof obj.ancestor === 'number' ? obj.ancestor : undefined; const sparse = readStringArray(obj, 'sparse'); - if (commit !== undefined && baseCommit !== undefined && commit !== baseCommit) { - throw new Error('workspace.repos[].commit and workspace.repos[].base_commit must match.'); - } - - if (!repoPath && !repo && !commit && !baseCommit && ancestor === undefined && !sparse) { + if (!repoPath && !repo && !commit && ancestor === undefined && !sparse) { return undefined; } @@ -69,7 +67,6 @@ export function parseRepoConfig(raw: unknown): RepoConfig | undefined { ...(repoPath !== undefined && { path: repoPath }), ...(repo !== undefined && { repo }), ...(commit !== undefined && { commit }), - ...(baseCommit !== undefined && { base_commit: baseCommit }), ...(ancestor !== undefined && { ancestor }), ...(sparse !== undefined && { sparse }), }; diff --git a/packages/core/test/evaluation/repo-schema-validation.test.ts b/packages/core/test/evaluation/repo-schema-validation.test.ts index 3c1756246..9d59486da 100644 --- a/packages/core/test/evaluation/repo-schema-validation.test.ts +++ b/packages/core/test/evaluation/repo-schema-validation.test.ts @@ -36,7 +36,7 @@ describe('repo lifecycle schema validation', () => { { path: './repo-a', repo: 'org/repo', - base_commit: '4a1b2c3d', + commit: '4a1b2c3d', }, ], }, @@ -49,7 +49,7 @@ describe('repo lifecycle schema validation', () => { ...baseEval, workspace: { docker: { image: 'swebench/sweb.eval.django__django:latest' }, - repos: [{ path: '/testbed', base_commit: 'abc123' }], + repos: [{ path: '/testbed', commit: 'abc123' }], }, }); expect(result.success).toBe(true); @@ -136,7 +136,7 @@ describe('repo lifecycle schema validation', () => { expect(result.success).toBe(false); }); - it('rejects conflicting commit aliases', () => { + it('rejects removed base_commit field', () => { const result = EvalFileSchema.safeParse({ ...baseEval, workspace: { @@ -144,7 +144,6 @@ describe('repo lifecycle schema validation', () => { { path: './repo-a', repo: 'https://github.com/org/repo.git', - commit: 'abc', base_commit: 'def', }, ], diff --git a/packages/core/test/evaluation/validation/eval-validator.test.ts b/packages/core/test/evaluation/validation/eval-validator.test.ts index 520dcb658..1e4766444 100644 --- a/packages/core/test/evaluation/validation/eval-validator.test.ts +++ b/packages/core/test/evaluation/validation/eval-validator.test.ts @@ -1980,7 +1980,7 @@ tests: image: swebench/sweb.eval.django__django:latest repos: - path: /testbed - base_commit: abc123 + commit: abc123 tests: - id: test-1 criteria: Goal @@ -1994,7 +1994,7 @@ tests: expect(result.errors.filter((e) => e.severity === 'error')).toHaveLength(0); }); - it('errors when commit aliases conflict', async () => { + it('errors when base_commit is used', async () => { const filePath = path.join(tempDir, 'workspace-conflicting-commits.yaml'); await writeFile( filePath, @@ -2002,7 +2002,6 @@ tests: repos: - path: ./repo repo: https://github.com/org/repo.git - commit: abc base_commit: def tests: - id: test-1 @@ -2016,7 +2015,9 @@ tests: expect(result.valid).toBe(false); expect( result.errors.some( - (e) => e.severity === 'error' && e.message.includes('commit and repos[].base_commit'), + (e) => + e.severity === 'error' && + e.message.includes('workspace.repos[].base_commit has been removed'), ), ).toBe(true); }); diff --git a/packages/core/test/evaluation/workspace-config-parsing.test.ts b/packages/core/test/evaluation/workspace-config-parsing.test.ts index ede3f0d75..58d577b34 100644 --- a/packages/core/test/evaluation/workspace-config-parsing.test.ts +++ b/packages/core/test/evaluation/workspace-config-parsing.test.ts @@ -61,7 +61,7 @@ tests: criteria: "Bug should be fixed" metadata: repo: sympy/sympy - base_commit: "abc123def" + source_commit: "abc123def" `, ); @@ -69,7 +69,7 @@ tests: expect(cases).toHaveLength(1); expect(cases[0].metadata).toEqual({ repo: 'sympy/sympy', - base_commit: 'abc123def', + source_commit: 'abc123def', }); }); @@ -234,7 +234,7 @@ tests: image: swebench/sweb.eval.django__django:latest repos: - path: /testbed - base_commit: abc123def + commit: abc123def `, ); @@ -246,7 +246,7 @@ tests: expect(cases[0].workspace?.repos).toHaveLength(1); expect(cases[0].workspace?.repos?.[0].path).toBe('/testbed'); expect(cases[0].workspace?.repos?.[0].repo).toBeUndefined(); - expect(cases[0].workspace?.repos?.[0].base_commit).toBe('abc123def'); + expect(cases[0].workspace?.repos?.[0].commit).toBe('abc123def'); }); it('should parse Docker repos with path + commit but no repo', async () => { @@ -274,7 +274,7 @@ tests: expect(cases[0].workspace?.repos?.[0].commit).toBe('v2.0.0'); }); - it('should parse repo base_commit alias', async () => { + it('rejects removed repo base_commit field', async () => { const evalFile = path.join(testDir, 'workspace-repo-base-commit.yaml'); await writeFile( evalFile, @@ -291,9 +291,9 @@ tests: `, ); - const cases = await loadTests(evalFile, testDir); - expect(cases).toHaveLength(1); - expect(cases[0].workspace?.repos?.[0].base_commit).toBe('abc123def'); + await expect(loadTests(evalFile, testDir)).rejects.toThrow( + 'workspace.repos[].base_commit has been removed', + ); }); it('parses workspace repos from YAML', async () => { diff --git a/packages/core/test/evaluation/workspace/deps-scanner.test.ts b/packages/core/test/evaluation/workspace/deps-scanner.test.ts index 163f8ffdf..b3d306c02 100644 --- a/packages/core/test/evaluation/workspace/deps-scanner.test.ts +++ b/packages/core/test/evaluation/workspace/deps-scanner.test.ts @@ -101,7 +101,7 @@ tests: }); }); - it('uses base_commit as a commit alias', async () => { + it('reports removed base_commit field', async () => { const file = await writeYaml( 'base-commit.eval.yaml', ` @@ -118,8 +118,9 @@ tests: ); const result = await scanRepoDeps([file]); - expect(result.errors).toHaveLength(0); - expect(result.repos[0].ref).toBe('abc123'); + expect(result.repos).toHaveLength(0); + expect(result.errors).toHaveLength(1); + expect(result.errors[0].message).toContain('workspace.repos[].base_commit has been removed'); }); it('deduplicates repos by canonical identity and ref', async () => { diff --git a/packages/core/test/evaluation/workspace/repo-manager.test.ts b/packages/core/test/evaluation/workspace/repo-manager.test.ts index 214a3fead..7e710eb01 100644 --- a/packages/core/test/evaluation/workspace/repo-manager.test.ts +++ b/packages/core/test/evaluation/workspace/repo-manager.test.ts @@ -619,7 +619,7 @@ describe('RepoManager', () => { expect(existsSync(path.join(targetDir, 'third.txt'))).toBe(false); }, 30_000); - it('checks out raw base_commit SHAs', async () => { + it('checks out raw commit SHAs', async () => { const repoDir = path.join(tmpDir, 'source-repo'); createTestRepo(repoDir); writeFileSync(path.join(repoDir, 'second.txt'), 'second'); @@ -635,7 +635,7 @@ describe('RepoManager', () => { { path: './my-repo', repo: `file://${remoteDir}`, - base_commit: secondSha, + commit: secondSha, }, workspaceDir, ); diff --git a/packages/core/test/evaluation/workspace/script-executor.test.ts b/packages/core/test/evaluation/workspace/script-executor.test.ts index e0d149c52..4efe68b09 100644 --- a/packages/core/test/evaluation/workspace/script-executor.test.ts +++ b/packages/core/test/evaluation/workspace/script-executor.test.ts @@ -212,7 +212,7 @@ rl.on('close', () => { testId: 'sympy-20590', evalRunId: 'run-123', caseInput: 'Fix the bug in issue #20590...', - caseMetadata: { repo: 'sympy/sympy', base_commit: '9aabb237' }, + caseMetadata: { repo: 'sympy/sympy', source_commit: '9aabb237' }, }; const output = await executeWorkspaceScript(config, context); diff --git a/packages/sdk/src/eval.ts b/packages/sdk/src/eval.ts index ee507a95c..06ec0c1e3 100644 --- a/packages/sdk/src/eval.ts +++ b/packages/sdk/src/eval.ts @@ -7,7 +7,6 @@ const KNOWN_SNAKE_CASE_KEYS = { afterAll: 'after_all', afterEach: 'after_each', argsMatch: 'args_match', - baseCommit: 'base_commit', beforeAll: 'before_all', beforeEach: 'before_each', budgetUsd: 'budget_usd', @@ -102,7 +101,6 @@ export interface EvalWorkspaceRepo { readonly path?: string; readonly repo?: string; readonly commit?: string; - readonly baseCommit?: string; readonly ancestor?: number; readonly sparse?: readonly string[]; } diff --git a/scripts/import-huggingface.py b/scripts/import-huggingface.py index b0d4c470f..7419036b0 100644 --- a/scripts/import-huggingface.py +++ b/scripts/import-huggingface.py @@ -23,7 +23,7 @@ problem_statement -> input (user message) repo -> metadata.repo instance_id -> workspace.docker.image (ghcr.io/epoch-research/swe-bench.eval.x86_64.:latest) - base_commit -> workspace.repos[].base_commit + base_commit -> workspace.repos[].commit FAIL_TO_PASS -> assertions (code-grader commands) difficulty -> metadata.difficulty @@ -191,7 +191,7 @@ def _convert_swebench_instance(row: dict[str, Any]) -> dict[str, Any]: } workspace: dict[str, Any] = {"docker": docker_config} if base_commit: - workspace["repos"] = [{"path": "/testbed", "base_commit": base_commit}] + workspace["repos"] = [{"path": "/testbed", "commit": base_commit}] eval_doc["workspace"] = workspace eval_doc["tests"] = [test_case] diff --git a/skills-data/agentv-bench/references/eval-yaml-spec.md b/skills-data/agentv-bench/references/eval-yaml-spec.md index 219ca0f7e..6b6303837 100644 --- a/skills-data/agentv-bench/references/eval-yaml-spec.md +++ b/skills-data/agentv-bench/references/eval-yaml-spec.md @@ -32,9 +32,9 @@ intentional grader panels. Write `expected_output` as a golden/reference answer, not as criteria or scoring instructions. For historical or repo-state evals, materialize the repository under -`workspace.repos[]` and pin `commit` or `base_commit` to the commit under test. -A SHA in prompt prose or metadata is context only; it does not give the agent an -actual checkout. +`workspace.repos[]` and pin `commit` to the commit under test. A SHA in prompt +prose or metadata is context only; it does not give the agent an actual +checkout. ## 2. Assertion Types and Grading Recipes diff --git a/skills-data/agentv-eval-migrations/references/breaking-changes.md b/skills-data/agentv-eval-migrations/references/breaking-changes.md index fa0317849..544fafdca 100644 --- a/skills-data/agentv-eval-migrations/references/breaking-changes.md +++ b/skills-data/agentv-eval-migrations/references/breaking-changes.md @@ -579,10 +579,11 @@ suffixes in `file://...:beforeAll`. ### v4.42.4 Shape -v4.42.4 already documented `workspace.repos[].repo`, `commit`, `ancestor`, and -`sparse` as repo provenance. It also accepted `base_commit` as a -SWE-bench-friendly alias for `commit`. The v4.42.4 parser rejected some -acquisition fields such as `source`, `checkout`, and `clone`. +v4.42.4 documented `workspace.repos[].repo`, `commit`, `base_commit`, +`ancestor`, and `sparse` as repo provenance. Treat `base_commit` as a legacy +authoring alias during migration; normalize it to `commit` instead of carrying +it forward. The v4.42.4 parser rejected some acquisition fields such as +`source`, `checkout`, and `clone`. ### Current Shape @@ -604,6 +605,7 @@ workspace: - `workspace.repos[].source` -> `workspace.repos[].repo`. - `workspace.repos[].checkout.ref` or similar -> `commit`. +- `workspace.repos[].base_commit` -> `commit`. - `workspace.repos[].clone.sparse` -> top-level `sparse`. - Remove `type`, `resolve`, and `resolver` from repo entries. - Configure acquisition policy in repo resolver/project config, not in eval @@ -613,7 +615,7 @@ workspace: ```bash bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml -rg -n "source:|checkout:|clone:|type:|resolve:|resolver:" path/to/evals +rg -n "source:|checkout:|clone:|base_commit:|type:|resolve:|resolver:" path/to/evals ``` Inspect `type:` matches manually because grader entries still use `type`. @@ -622,9 +624,9 @@ Inspect `type:` matches manually because grader entries still use `type`. `repos[].path` remains valid and means the target directory inside the materialized workspace. It is not the removed local `workspace.path`. -`base_commit` remains accepted as an alias for `commit`, mainly for -SWE-bench-style datasets, but migrated examples should use `commit` unless the -source dataset convention specifically uses `base_commit`. +Current parser rejects `base_commit`. Prefer a single checkout pin field, +`commit`, so workers do not preserve a second spelling in examples or generated +evals. ## Target And Runtime Separation diff --git a/skills-data/agentv-eval-review/SKILL.md b/skills-data/agentv-eval-review/SKILL.md index 369b36db6..f3f5ddbb5 100644 --- a/skills-data/agentv-eval-review/SKILL.md +++ b/skills-data/agentv-eval-review/SKILL.md @@ -27,7 +27,7 @@ Walk every target eval file and report violations grouped by severity (error > w - Flag `criteria` that duplicates assertion strings when `assertions` already express the grading contract (warning — remove the duplicate `criteria`). - Prefer plain assertion strings over multiple named `type: llm-rubric` blocks when the default LLM rubric grader can evaluate the checks (info unless custom prompts or grader targets are present). - Detect `expected_output` prose patterns like "The agent should..." or "The output is..." (warning — `expected_output` should be a golden/reference answer; scoring rules belong in `assertions` or, for implicit-grader cases, `criteria`). -- For historical or repo-state evals, verify the relevant repo is pinned under `workspace.repos[].commit` or `workspace.repos[].base_commit`; a SHA mentioned only in prompt prose or metadata is not an operational checkout (warning). +- For historical or repo-state evals, verify the relevant repo is pinned under `workspace.repos[].commit`; a SHA mentioned only in prompt prose or metadata is not an operational checkout (warning). - Identical file inputs repeated across multiple tests in the same eval should be hoisted to a top-level `input` (info). - Eval files in the same directory should share a common `id` prefix (info — flag drift). diff --git a/skills-data/agentv-eval-writer/SKILL.md b/skills-data/agentv-eval-writer/SKILL.md index b4d3d1bfd..0b26b596e 100644 --- a/skills-data/agentv-eval-writer/SKILL.md +++ b/skills-data/agentv-eval-writer/SKILL.md @@ -336,7 +336,7 @@ workspace: repos: - path: ./repo repo: sympy/sympy - base_commit: "abc123" + commit: "abc123" hooks: before_all: command: ["bun", "run", "setup.ts"] @@ -358,10 +358,10 @@ tests: **Merge:** Case-level fields replace suite-level fields. **Commands receive stdin JSON:** `{workspace_path, test_id, eval_run_id, case_input, case_metadata}` **Setup failure:** aborts case. **Teardown failure:** non-fatal (warning). -For SWE-bench-style evals, keep operational checkout state under `workspace.repos[].base_commit`; treat `metadata.source_commit` as informational only. -For historical repo-state evals, pin `workspace.repos[].commit` or -`workspace.repos[].base_commit` to the commit under test. A SHA in the prompt or -metadata without a matching workspace repo pin is not an operational checkout. +For SWE-bench-style evals, translate the upstream `base_commit` value to +`workspace.repos[].commit`; treat `metadata.source_commit` as informational +only. A SHA in the prompt or metadata without a matching workspace repo pin is +not an operational checkout. ### Repository Lifecycle @@ -372,7 +372,6 @@ workspace: repos: - path: ./repo repo: https://github.com/org/repo.git - resolver: org_snapshots # optional repo_resolvers[].name override commit: main ancestor: 1 # parent commit hooks: @@ -382,12 +381,10 @@ workspace: ``` - `repo`: full clone URL or GitHub `org/name` shorthand -- `resolver`: optional configured repo resolver name; omit for pattern/default/built-in selection - `commit`: branch, tag, or SHA to check out -- `base_commit`: alias for `commit` for SWE-bench-style datasets - `ancestor`: walk N commits back from the checked-out ref - `sparse`: sparse checkout paths array -- Do not use legacy `source`, `type`, `checkout`, `resolve`, or `clone` fields under `workspace.repos[]` +- Do not use legacy `source`, `type`, `checkout`, `base_commit`, `resolve`, `resolver`, or `clone` fields under `workspace.repos[]` - Do not author `workspace.mode`, `workspace.path`, `experiment.workspace`, or `execution.workspace` in eval YAML - Harness-managed repo workspaces use temp materialization by default; use `workspace.scope: suite | attempt` for portable lifetime - Existing local workspace directories are machine-local bindings; use `--workspace-path` or `.agentv/config.local.yaml` with `execution.workspace_path` diff --git a/skills-data/agentv-eval-writer/references/eval.schema.json b/skills-data/agentv-eval-writer/references/eval.schema.json index a80f4f1dc..f92b7164a 100644 --- a/skills-data/agentv-eval-writer/references/eval.schema.json +++ b/skills-data/agentv-eval-writer/references/eval.schema.json @@ -2433,10 +2433,6 @@ "type": "string", "minLength": 1 }, - "base_commit": { - "type": "string", - "minLength": 1 - }, "ancestor": { "type": "integer", "minimum": 0 @@ -4473,10 +4469,6 @@ "type": "string", "minLength": 1 }, - "base_commit": { - "type": "string", - "minLength": 1 - }, "ancestor": { "type": "integer", "minimum": 0 @@ -10237,10 +10229,6 @@ "type": "string", "minLength": 1 }, - "base_commit": { - "type": "string", - "minLength": 1 - }, "ancestor": { "type": "integer", "minimum": 0 @@ -10885,10 +10873,6 @@ "type": "string", "minLength": 1 }, - "base_commit": { - "type": "string", - "minLength": 1 - }, "ancestor": { "type": "integer", "minimum": 0 From 7c4d53464390e2c75d57e176eaa80869b79812c9 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Fri, 3 Jul 2026 11:09:14 +0200 Subject: [PATCH 4/4] fix(eval): treat repo aliases as unsupported fields --- .../next/guides/workspace-architecture.mdx | 3 +- .../content/docs/docs/next/tools/import.mdx | 2 +- .../evaluation/validation/eval-validator.ts | 24 ++++++++-------- .../workspace/repo-config-parser.ts | 12 ++++++-- .../evaluation/repo-schema-validation.test.ts | 16 ----------- .../validation/eval-validator.test.ts | 28 ------------------- .../workspace-config-parsing.test.ts | 22 --------------- .../evaluation/workspace/deps-scanner.test.ts | 8 +++--- .../workspace/script-executor.test.ts | 2 +- .../references/breaking-changes.md | 5 ++-- skills-data/agentv-eval-writer/SKILL.md | 4 +-- 11 files changed, 32 insertions(+), 94 deletions(-) diff --git a/apps/web/src/content/docs/docs/next/guides/workspace-architecture.mdx b/apps/web/src/content/docs/docs/next/guides/workspace-architecture.mdx index d954334c9..4bcda0a0a 100644 --- a/apps/web/src/content/docs/docs/next/guides/workspace-architecture.mdx +++ b/apps/web/src/content/docs/docs/next/guides/workspace-architecture.mdx @@ -88,8 +88,7 @@ Supported repo fields: | `sparse` | Optional sparse-checkout paths | | `ancestor` | Walk N parents back after resolving `commit` | -`commit` is the AgentV checkout pin. Translate upstream dataset columns such as -SWE-bench `base_commit` to `commit` when importing eval YAML. +`commit` is the AgentV checkout pin. `source`, `type`, `checkout`, `checkout.resolve`, `clone`, `resolve`, and `resolver` are not part of the repo schema. Acquisition settings are diff --git a/apps/web/src/content/docs/docs/next/tools/import.mdx b/apps/web/src/content/docs/docs/next/tools/import.mdx index cd5a6acef..6184971e0 100644 --- a/apps/web/src/content/docs/docs/next/tools/import.mdx +++ b/apps/web/src/content/docs/docs/next/tools/import.mdx @@ -218,7 +218,7 @@ uv run scripts/import-huggingface.py \ Each instance becomes an EVAL.yaml with: - `input` — the problem statement - `workspace.docker.image` — the pre-built SWE-bench Docker image (`ghcr.io/epoch-research/swe-bench.eval.x86_64.:latest`) -- `workspace.repos[].commit` — the SWE-bench `base_commit` value to reset to before the agent runs +- `workspace.repos[].commit` — the commit to reset to before the agent runs - `assertions` — `script` tasks that run `FAIL_TO_PASS` and `PASS_TO_PASS` pytest suites inside the container Run an imported SWE-bench eval against any coding agent target: diff --git a/packages/core/src/evaluation/validation/eval-validator.ts b/packages/core/src/evaluation/validation/eval-validator.ts index f293fdd84..750f9b6e1 100644 --- a/packages/core/src/evaluation/validation/eval-validator.ts +++ b/packages/core/src/evaluation/validation/eval-validator.ts @@ -229,6 +229,8 @@ const KNOWN_TEST_FIELDS = new Set([ 'window_size', ]); +const SUPPORTED_WORKSPACE_REPO_FIELDS = new Set(['path', 'repo', 'commit', 'ancestor', 'sparse']); + /** Removed test-level fields with migration hints. */ const REMOVED_TEST_FIELDS = new Map([]); @@ -1637,16 +1639,6 @@ function validateWorkspaceRepoConfig( }); } - if ('base_commit' in repo) { - errors.push({ - severity: 'error', - filePath, - location: `${location}.repos[path=${repo.path ?? '(none)'}]`, - message: - 'workspace.repos[].base_commit has been removed. Use workspace.repos[].commit.', - }); - } - if ('clone' in repo) { errors.push({ severity: 'error', @@ -1684,6 +1676,17 @@ function validateWorkspaceRepoConfig( }); } + for (const key of Object.keys(repo)) { + if (!SUPPORTED_WORKSPACE_REPO_FIELDS.has(key)) { + errors.push({ + severity: 'error', + filePath, + location: `${location}.repos[path=${repo.path ?? '(none)'}]`, + message: `workspace.repos[].${key} is not supported. Supported fields: path, repo, commit, ancestor, sparse.`, + }); + } + } + if (!repo.repo && !isObject(docker)) { errors.push({ severity: 'error', @@ -1694,7 +1697,6 @@ function validateWorkspaceRepoConfig( 'Repo-less entries are only valid when workspace.docker is configured.', }); } - } } diff --git a/packages/core/src/evaluation/workspace/repo-config-parser.ts b/packages/core/src/evaluation/workspace/repo-config-parser.ts index 7a88051cc..f8dbf7c0b 100644 --- a/packages/core/src/evaluation/workspace/repo-config-parser.ts +++ b/packages/core/src/evaluation/workspace/repo-config-parser.ts @@ -9,6 +9,8 @@ import type { RepoConfig } from '../types.js'; import { isJsonObject } from '../types.js'; +const SUPPORTED_REPO_FIELDS = new Set(['path', 'repo', 'commit', 'ancestor', 'sparse']); + function readString(obj: Record, key: string): string | undefined { const value = obj[key]; return typeof value === 'string' && value.trim().length > 0 ? value : undefined; @@ -33,9 +35,6 @@ export function parseRepoConfig(raw: unknown): RepoConfig | undefined { 'workspace.repos[].checkout has been removed. Use top-level commit and ancestor.', ); } - if ('base_commit' in obj) { - throw new Error('workspace.repos[].base_commit has been removed. Use workspace.repos[].commit.'); - } if ('clone' in obj) { throw new Error('workspace.repos[].clone has been removed. Use top-level sparse if needed.'); } @@ -52,6 +51,13 @@ export function parseRepoConfig(raw: unknown): RepoConfig | undefined { 'workspace.repos[].resolver has been removed. Configure repo_resolvers.repos patterns instead.', ); } + for (const key of Object.keys(obj)) { + if (!SUPPORTED_REPO_FIELDS.has(key)) { + throw new Error( + `workspace.repos[].${key} is not supported. Supported fields: path, repo, commit, ancestor, sparse.`, + ); + } + } const repoPath = readString(obj, 'path'); const repo = readString(obj, 'repo'); diff --git a/packages/core/test/evaluation/repo-schema-validation.test.ts b/packages/core/test/evaluation/repo-schema-validation.test.ts index 9d59486da..aa68d39be 100644 --- a/packages/core/test/evaluation/repo-schema-validation.test.ts +++ b/packages/core/test/evaluation/repo-schema-validation.test.ts @@ -136,22 +136,6 @@ describe('repo lifecycle schema validation', () => { expect(result.success).toBe(false); }); - it('rejects removed base_commit field', () => { - const result = EvalFileSchema.safeParse({ - ...baseEval, - workspace: { - repos: [ - { - path: './repo-a', - repo: 'https://github.com/org/repo.git', - base_commit: 'def', - }, - ], - }, - }); - expect(result.success).toBe(false); - }); - it('accepts workspace with hooks after_each reset config', () => { const result = EvalFileSchema.safeParse({ ...baseEval, diff --git a/packages/core/test/evaluation/validation/eval-validator.test.ts b/packages/core/test/evaluation/validation/eval-validator.test.ts index 1e4766444..d0bb9d27a 100644 --- a/packages/core/test/evaluation/validation/eval-validator.test.ts +++ b/packages/core/test/evaluation/validation/eval-validator.test.ts @@ -1994,34 +1994,6 @@ tests: expect(result.errors.filter((e) => e.severity === 'error')).toHaveLength(0); }); - it('errors when base_commit is used', async () => { - const filePath = path.join(tempDir, 'workspace-conflicting-commits.yaml'); - await writeFile( - filePath, - `workspace: - repos: - - path: ./repo - repo: https://github.com/org/repo.git - base_commit: def -tests: - - id: test-1 - criteria: Goal - input: "Query" -`, - ); - - const result = await validateEvalFile(filePath); - - expect(result.valid).toBe(false); - expect( - result.errors.some( - (e) => - e.severity === 'error' && - e.message.includes('workspace.repos[].base_commit has been removed'), - ), - ).toBe(true); - }); - it('errors when an external workspace file uses legacy source', async () => { const workspaceFile = path.join(tempDir, 'external-workspace.yaml'); await writeFile( diff --git a/packages/core/test/evaluation/workspace-config-parsing.test.ts b/packages/core/test/evaluation/workspace-config-parsing.test.ts index 58d577b34..78d61f2aa 100644 --- a/packages/core/test/evaluation/workspace-config-parsing.test.ts +++ b/packages/core/test/evaluation/workspace-config-parsing.test.ts @@ -274,28 +274,6 @@ tests: expect(cases[0].workspace?.repos?.[0].commit).toBe('v2.0.0'); }); - it('rejects removed repo base_commit field', async () => { - const evalFile = path.join(testDir, 'workspace-repo-base-commit.yaml'); - await writeFile( - evalFile, - ` -tests: - - id: repo-base-commit - input: "Do something" - criteria: "Should work" - workspace: - repos: - - path: /testbed - repo: https://github.com/org/repo.git - base_commit: abc123def -`, - ); - - await expect(loadTests(evalFile, testDir)).rejects.toThrow( - 'workspace.repos[].base_commit has been removed', - ); - }); - it('parses workspace repos from YAML', async () => { const evalFile = path.join(testDir, 'workspace-repos.yaml'); await writeFile( diff --git a/packages/core/test/evaluation/workspace/deps-scanner.test.ts b/packages/core/test/evaluation/workspace/deps-scanner.test.ts index b3d306c02..61c425f35 100644 --- a/packages/core/test/evaluation/workspace/deps-scanner.test.ts +++ b/packages/core/test/evaluation/workspace/deps-scanner.test.ts @@ -101,15 +101,15 @@ tests: }); }); - it('reports removed base_commit field', async () => { + it('reports unsupported repo fields', async () => { const file = await writeYaml( - 'base-commit.eval.yaml', + 'unsupported-repo-field.eval.yaml', ` workspace: repos: - path: ./repo repo: https://github.com/org/repo.git - base_commit: abc123 + unknown_field: abc123 tests: - id: test-1 input: hello @@ -120,7 +120,7 @@ tests: const result = await scanRepoDeps([file]); expect(result.repos).toHaveLength(0); expect(result.errors).toHaveLength(1); - expect(result.errors[0].message).toContain('workspace.repos[].base_commit has been removed'); + expect(result.errors[0].message).toContain('workspace.repos[].unknown_field is not supported'); }); it('deduplicates repos by canonical identity and ref', async () => { diff --git a/packages/core/test/evaluation/workspace/script-executor.test.ts b/packages/core/test/evaluation/workspace/script-executor.test.ts index 4efe68b09..f32b7aed6 100644 --- a/packages/core/test/evaluation/workspace/script-executor.test.ts +++ b/packages/core/test/evaluation/workspace/script-executor.test.ts @@ -212,7 +212,7 @@ rl.on('close', () => { testId: 'sympy-20590', evalRunId: 'run-123', caseInput: 'Fix the bug in issue #20590...', - caseMetadata: { repo: 'sympy/sympy', source_commit: '9aabb237' }, + caseMetadata: { repo: 'sympy/sympy' }, }; const output = await executeWorkspaceScript(config, context); diff --git a/skills-data/agentv-eval-migrations/references/breaking-changes.md b/skills-data/agentv-eval-migrations/references/breaking-changes.md index 544fafdca..113cdb7b6 100644 --- a/skills-data/agentv-eval-migrations/references/breaking-changes.md +++ b/skills-data/agentv-eval-migrations/references/breaking-changes.md @@ -624,9 +624,8 @@ Inspect `type:` matches manually because grader entries still use `type`. `repos[].path` remains valid and means the target directory inside the materialized workspace. It is not the removed local `workspace.path`. -Current parser rejects `base_commit`. Prefer a single checkout pin field, -`commit`, so workers do not preserve a second spelling in examples or generated -evals. +Current eval YAML accepts a single checkout pin field, `commit`, so workers +should not preserve a second spelling in examples or generated evals. ## Target And Runtime Separation diff --git a/skills-data/agentv-eval-writer/SKILL.md b/skills-data/agentv-eval-writer/SKILL.md index 0b26b596e..9e7e1171a 100644 --- a/skills-data/agentv-eval-writer/SKILL.md +++ b/skills-data/agentv-eval-writer/SKILL.md @@ -358,7 +358,7 @@ tests: **Merge:** Case-level fields replace suite-level fields. **Commands receive stdin JSON:** `{workspace_path, test_id, eval_run_id, case_input, case_metadata}` **Setup failure:** aborts case. **Teardown failure:** non-fatal (warning). -For SWE-bench-style evals, translate the upstream `base_commit` value to +For SWE-bench-style evals, put operational checkout state under `workspace.repos[].commit`; treat `metadata.source_commit` as informational only. A SHA in the prompt or metadata without a matching workspace repo pin is not an operational checkout. @@ -384,8 +384,6 @@ workspace: - `commit`: branch, tag, or SHA to check out - `ancestor`: walk N commits back from the checked-out ref - `sparse`: sparse checkout paths array -- Do not use legacy `source`, `type`, `checkout`, `base_commit`, `resolve`, `resolver`, or `clone` fields under `workspace.repos[]` -- Do not author `workspace.mode`, `workspace.path`, `experiment.workspace`, or `execution.workspace` in eval YAML - Harness-managed repo workspaces use temp materialization by default; use `workspace.scope: suite | attempt` for portable lifetime - Existing local workspace directories are machine-local bindings; use `--workspace-path` or `.agentv/config.local.yaml` with `execution.workspace_path` - `hooks.enabled`: boolean (default `true`); set `false` to skip all lifecycle hooks