Skip to content

Add Opus 4.8 and DeepSeek V4 Flash leaderboard runs #21

Description

@ShenzheZhu

Target branch: main

Classification: experiment refresh / leaderboard update

Problem statement:
The public A2A-NT leaderboard needs two additional frontier models: anthropic/claude-opus-4.8 and deepseek/deepseek-v4-flash.

Expected behavior:
The sweep config includes both models, the runner can execute an incremental sweep for pairs touching newly added models, and the resulting analysis can be merged into the current leaderboard data without rerunning completed legacy pairs.

Scope:

  • Add the two frontier model entries to configs/sweep_example.json.
  • Add or use a scoped run path that only runs pairs involving these new models.
  • Validate model IDs against OpenRouter before live calls.
  • Run a small pilot before launching the full incremental run.
  • Regenerate summary and web leaderboard data after completion.

Acceptance criteria:

  • Dry run shows the expected incremental pair count.
  • Pilot results complete without system/provider errors.
  • Full incremental run writes a manifest and resumes safely with append semantics.
  • Leaderboard data includes the two new models after postprocessing.

Out of scope:

  • Changing benchmark scoring definitions.
  • Re-running already completed old-model cells unless required for repair.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions