Target branch: main
Classification: experiment refresh / leaderboard update
Problem statement:
The public A2A-NT leaderboard needs two additional frontier models: anthropic/claude-opus-4.8 and deepseek/deepseek-v4-flash.
Expected behavior:
The sweep config includes both models, the runner can execute an incremental sweep for pairs touching newly added models, and the resulting analysis can be merged into the current leaderboard data without rerunning completed legacy pairs.
Scope:
- Add the two frontier model entries to configs/sweep_example.json.
- Add or use a scoped run path that only runs pairs involving these new models.
- Validate model IDs against OpenRouter before live calls.
- Run a small pilot before launching the full incremental run.
- Regenerate summary and web leaderboard data after completion.
Acceptance criteria:
- Dry run shows the expected incremental pair count.
- Pilot results complete without system/provider errors.
- Full incremental run writes a manifest and resumes safely with append semantics.
- Leaderboard data includes the two new models after postprocessing.
Out of scope:
- Changing benchmark scoring definitions.
- Re-running already completed old-model cells unless required for repair.
Target branch: main
Classification: experiment refresh / leaderboard update
Problem statement:
The public A2A-NT leaderboard needs two additional frontier models: anthropic/claude-opus-4.8 and deepseek/deepseek-v4-flash.
Expected behavior:
The sweep config includes both models, the runner can execute an incremental sweep for pairs touching newly added models, and the resulting analysis can be merged into the current leaderboard data without rerunning completed legacy pairs.
Scope:
Acceptance criteria:
Out of scope: