eval: include structured results, run config, and summary in JSON output

[hamza-jeddad]

What

The eval JSON output (<run-name>.json) now includes everything that was previously only in the log file:

• Run metadata: name, timestamp, duration, config (agent, judge model, concurrency, evals dir), and aggregate summary
• Per-session eval_result with overall pass/fail, human-readable successes/failures, error, cost, output tokens
• Structured checks: size (actual vs expected), tool_calls (F1 score), and relevance (per-criterion pass/fail with judge reasoning)
• Judge reasons for ALL criteria — both passed and failed. The LLM judge was already generating reasons for passed criteria but the code was discarding them.

JSON output format change

The output changes from a bare session array to a RunOutput wrapper:

{
  "name": "happy-panda-1234",
  "timestamp": "...",
  "duration": "42s",
  "config": { "agent": "...", "judge_model": "...", "concurrency": 8, "evals_dir": "..." },
  "summary": { "total_evals": 5, ... },
  "sessions": [
    {
      "title": "LedgerLite Container",
      "evals": { ... },
      "eval_result": {
        "passed": true,
        "successes": ["relevance 5/5"],
        "checks": {
          "relevance": {
            "passed": true,
            "passed_count": 5,
            "total": 5,
            "results": [
              { "criterion": "Uses app.py as entrypoint", "passed": true, "reason": "..." },
              { "criterion": "Keeps sample data optional", "passed": false, "reason": "..." }
            ]
          }
        }
      },
      "messages": [...]
    }
  ]
}

Files changed

• pkg/session/session.go — New types: EvalResult, EvalResultChecks, SizeCheck, ToolCallsCheck, RelevanceCheck, RelevanceCriterionResult
• pkg/evaluation/types.go — RunOutput, RunOutputConfig, Config field on EvalRun
• pkg/evaluation/judge.go — CheckRelevance now returns []RelevanceResult (all criteria with reasons) instead of (passed int, failed []RelevanceResult)
• pkg/evaluation/save.go — populateEvalResult + updated SaveRunSessionsJSON
• pkg/evaluation/eval.go — Updated CheckRelevance caller, set Config on EvalRun
• Tests updated accordingly

Notes

• No changes to the session DB schema — eval_result only appears in the JSON output
• This is a breaking change to the JSON output format (array → object wrapper)

[docker-agent]

Assessment: 🔴 CRITICAL

Found 1 critical data integrity issue in the relevance checking error handling.

[hamza-jeddad]

/review

[docker-agent]

Assessment: 🟢 APPROVE

No issues found in the changed code. The PR properly refactors the eval JSON output format to include structured results, run configuration, and summary metadata. All changes are well-tested and error handling is appropriate.

Key changes reviewed:

• RunOutput wrapper structure with metadata, config, and summary
• CheckRelevance now returns results for all criteria (passed and failed) with judge reasoning
• Proper updates to all call sites and comprehensive test coverage
• Breaking change to JSON output format is documented in PR description