Skip to content

AGENTS.md

Latest commit

 

History

History
51 lines (41 loc) · 6.36 KB

AGENTS.md

File metadata and controls

51 lines (41 loc) · 6.36 KB
Raw

benchflow

Multi-turn agent benchmarking with ACP. Docs live in docs/.

Setup + test

uv sync --extra dev --locked
uv run python -m pytest tests/
uv run ty check src/
uv run ruff check .

Conventions

  • Don't rewrite passing tests to match new behavior. Update for shape changes, not for semantic changes you don't understand. No tautological tests.
  • Regression tests must name the PR/commit they guard in the docstring (e.g. Guards the fix from PR #198 against the regression introduced by PR #193).
  • Human review before main. PRs only. No force-pushes to main. Self-approval doesn't count.
  • Trunk-based: branch off main, PR back to main. No long-lived release branches.
  • Releases: current mechanics live in docs/release.md. Merges to main publish internal preview .devN builds after CI passes; public releases require a reviewed stable-version PR, a matching v<version> tag on main, then a bump back to the next .dev0.

Experiment guidance (when using benchflow to run batch tasks experiments)

  • When running tasks experiments on Daytona, keep in mind that Daytona supports up to 100 parallel VMs.
  • When running tasks experiments by default use the highest inference level for all the model calls (For example, xhigh for GPT5.5, or MAX for Opus4.8).
  • When running tasks experiments, for each API key that available to you, verify 1. whether the key is still working by sending basic LLM calls; 2. ask the user how many parallel tasks can each key support / figure it out by yourself. Thus you can use highest available concurrency when running experiments.
  • A revoked/wrong model key shows up as an opaque agent error, not an auth error. With openhands, a dead key surfaces only as ACP error -32603: Internal error on the first model call (and --usage-tracking off / --agent-env LLM_CACHING_PROMPT=false do not help). So before any run, verify the exact key string with a raw provider call: Gemini curl ".../v1beta/models/<model>:generateContent?key=$GEMINI_API_KEY" must return 200; Bedrock a converse call with Authorization: Bearer $AWS_BEARER_TOKEN_BEDROCK must return 200. Keys of the same provider can rotate format (e.g. Google AI Studio AQ.… vs legacy AIza…) — only the live one works, so never assume a .env/keys.env entry is current.
  • Daytona hard-caps each sandbox at 10 GB (a larger storage request is clamped — see the Clamping storage_mb … -> 10240 log line). Tasks with heavy images — large HuggingFace model snapshots, Playwright, LaTeX/marker, e.g. latex-formula-extraction — overflow during bootstrap and fail with No space left on device, or hang silently at "Sandbox user agent ready" with no trajectory (this is an infra/disk failure, not a model or auth bug). On Daytona pick light tasks (e.g. citation-check, 3d-scan-calc); for heavy tasks use --sandbox docker (host disk, no 10 GB cap).
  • Opus-4.8 (and other Claude 4.8+) on Bedrock needs the adaptive-thinking patch. Without it the first call 400s (thinking.type.enabled is not supported … use thinking.type.adaptive). It ships in the LiteLLM proxy as src/benchflow/providers/litellm_bedrock_patch.py (loaded into the proxy process via sitecustomize), so it applies the same way on both backends — there is no separate host Bedrock proxy. Model string is aws-bedrock/us.anthropic.claude-opus-4-8 with AWS_REGION=us-west-2. The run defaults to adaptive high; current LiteLLM Bedrock support also clamps requested xhigh/max to high for Claude 4.8 models. Verify the applied value in trajectory/llm_trajectory.jsonl rather than trusting a requested-effort label.
  • Use Huggingface as the ground truth of tasks traj database. The target is making sure number_of_tasks * 2 (with/without skills) * number_of_trials (usually 3 / 5) for each model + each harness is complete. Each traj on Huggingface must be "healthy" - "healthy" means 1. the trial has complete traj information (agent pass / fail / timeout the task); 2. the trial has complete meta information (for example: token usage; timing; etc.). For each traj, we should use subagents to audit 1. whether the fail / timeout is due to agent capability instead of task config or env setting; 2. there is no reward hacking behavior; 3. in "no-skill" trials the agents do not have access to any skill files or accessed any skill files in traj.
  • When reviewing task-run result trajectories, agents must use the installed benchflow-experiment-review skill from the active harness's skill root; in this repo the canonical copy lives at .agents/skills/benchflow-experiment-review, with .claude/skills kept as a symlink for Claude Code compatibility.

Skill catalog (.agents/skills, mirrored at .claude/skills)

Skill Use it for
benchflow Running benchmarks, creating tasks, inspecting results via the CLI/SDK.
benchflow-experiment-review Auditing already-published run trajectories — traj health, reward hacking, no-skill leakage, Daytona/Docker parity. Wired into .github/scripts/codex_review.py and integration-final-review.yml.
benchflow-traj-upload Uploading trajectories to the HuggingFace/GCS traj database.
skill-creator Creating, improving, evaluating, and packaging reusable skills. Vendored from anthropics/skills@f6656c1.
task-creator Authoring a new SkillsBench task, idea → submission-ready PR.
task-review Reviewing a SkillsBench task PR pre-merge — track routing, policy checks, oracle+agent benchmarks, report bundle.
thermo-nuclear-code-quality-review Deliberately harsh maintainability review of a branch. User-invoked only (disable-model-invocation: true).

task-creator and task-review are verbatim copies from benchflow-ai/skillsbench; they operate on a skillsbench checkout, not on this repo. Re-sync by copying .agents/skills/{task-creator,task-review} from there. skill-creator is a verbatim copy from anthropics/skills@f6656c1. Keep the two review skills distinct: task-review is pre-merge task-PR review, benchflow-experiment-review is post-run trajectory auditing.