Skip to content

benchmark.md

Latest commit

 

History

History
110 lines (82 loc) · 3.63 KB

benchmark.md

File metadata and controls

110 lines (82 loc) · 3.63 KB
Raw

Benchmark Suite

SDYJ benchmarks are designed as regression checks for agent behavior, not as one-off demos. A benchmark run should answer three questions:

  1. Did the agent produce a useful answer?
  2. Was the answer grounded in traceable evidence?
  3. Can failures be diagnosed from the trace?

Run

Offline benchmark with canned evidence and fake LLM:

python main.py benchmark run --max-scenarios 1 --max-iterations 2

Use it as a CI gate:

python main.py benchmark run --fail-under 0.75

Override individual metric thresholds:

python main.py benchmark run \
  --threshold trace_completeness=0.9 \
  --threshold citation_id_coverage=0.8

Check deterministic behavior by repeating offline runs:

python main.py benchmark run --determinism-repeats 2

Compare against a saved summary:

python main.py benchmark run --compare-summary outputs/eval_reports/eval_summary_YYYYMMDD_HHMMSS.json
python main.py benchmark compare baseline.json candidate.json

The older command remains supported:

python main.py eval --max-scenarios 1

Current Scenarios

Scenario What it tests
agent_reliability_hard RAG, web search, MCP-style tools, human approval, ablation, latency/cost thresholds
tool_failure_recovery_hard Timeout, empty result, duplicate URLs, fallback, source quality, review strategy
mcp_rag_ops_hard Customer-support MCP+RAG with privacy, refusal, safety boundaries, SLO, and cost budget
llm_failure_recovery_hard Injected transient + permanent LLM failures: gates require ≥1 recorded retry AND ≥1 recorded degradation while the report still ships

Metrics

Metric Meaning
plan_coverage Required concepts covered by plan/report
section_completeness Expected report sections present
citation_id_coverage Evidence IDs cited in the report BODY (reference list excluded — it enumerates every id by construction)
citation_validity_rate [E#] mentions that reference real evidence; fabricated ids lower it
invalid_citation_count Fabricated mentions surviving in the final report (expected 0)
citation_density_per_1k_chars Citation density normalized by length
tool_success_rate Retrieval batches without errors
grounded_key_finding_rate Key findings carrying a valid evidence citation
faithfulness_score LLM-judged claim↔evidence support; own threshold (0.70), NOT part of overall_score
citation_precision LLM-judged share of claims at least partially supported
retries_total Transient LLM failures recovered by retry
degraded_event_count Recorded graceful degradations
trace_completeness Trace has required fields, nodes, events, tool/LLM details, and replay cache
overall_score Compact dashboard score (deterministic metrics only, incl. validity)

The judge is dual-track: offline runs grade with a canned verdict (CI stays deterministic and key-free), --live grades with the real provider.

Each scenario can define thresholds. A benchmark summary includes:

  • passed
  • failed_scenarios
  • failed_thresholds
  • determinism
  • comparison
  • paths to generated report and trace artifacts

Live Mode

Run a real model while keeping canned evidence stable:

python main.py benchmark run \
  --live \
  --provider deepseek \
  --model deepseek-v4-flash \
  --scenario agent_reliability_hard

Also evaluate real retrieval:

python main.py benchmark run --live --live-search

Live-search scores can drift because external search results change. Use them as smoke tests, not as deterministic regression gates.