Pipeline: TE/AEGIS/TRAIL validator+confirmer fold-in, TRAIL LoCc metrics, WW refactor#10
Open
barakhsin wants to merge 14 commits into
Open
Pipeline: TE/AEGIS/TRAIL validator+confirmer fold-in, TRAIL LoCc metrics, WW refactor#10barakhsin wants to merge 14 commits into
barakhsin wants to merge 14 commits into
Conversation
Mirrors the agentrx example so the MASeval multi-evaluator judge +
EvidenceVerifier + report aggregation run on the TraceElephant failure-
attribution benchmark (arXiv:2604.22708, ACL 2026; 220 failure traces from
Captain-Agent / Magentic-One / SWE-Agent), scored against the gold
mistake_agent / mistake_step.
- trace_elephant_data.py: adapter walking data/{system}-runs-*/{task}/,
normalizing both on-disk shapes into {name, content} steps; 1-based step
index == gold mistake_step.
- launch_findings_judges.py: 11 LLM evaluators + EvidenceVerifier + report +
FinalAnswerVerifier in no-ground-truth mode (df="trace_elephant"; gt=None
routes to the MAS Task Completion judge, so no new extractor is needed).
- calculate_agent_step_accuracy.py: Who&When-parity agent/step scorer.
- verifier_ablation.py: none/strict/soft EvidenceVerifier ablation.
Data (data.zip) and .env are git-ignored; findings are generated locally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fold the deterministic non_llm_validators (+ LLM confirmation) into the
verifier=none judge predictions on TraceElephant, mirroring the Who&When
score_none_plus_validators analysis.
- gen_validators_confirm.py: run validators (+confirmer, or --no-confirm for the
free pass) into trace_elephant_{system}_valconfirm/; resumable.
- score_none_plus_validators.py: fold-in scorer. Shifts validator loci +1
(validators are 0-based via who_and_when_to_spans; judges+gold are 1-based via
format_trace start=1) so all four sources share one index space — this was the
entire "misaligned spans" reason the launcher skipped these validators, a narrow
bookkeeping offset, not a deeper incompatibility. Fold-in agent = step label at
the locus (sub-agent for captain/magentic, tool for swe). Reports surface vs
appointed locus and all four verdict filters, per-system + true overall.
- score_swe_tool_attribution.py: SWE culprit scored in the tool namespace (gold
mistake_agent is a tool == label of the gold step).
- llm_confirm.py: two-knob confirmer (full reading view + windowed appointing).
Result (117 finding-bearing traces; confirmed column, surface locus, overall):
Agent Hit 75.0->84.1, Step Hit 44.5->50.9; SWE Agent Hit 75.0->90.9.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tion Apply the Who&When/TraceElephant approach to AEGIS. - gen_validators_confirm.py: attach the LLM confirmation layer to AEGIS's already-stored non_llm_validators. Reloads each bearing trace's history from the source JSONL matched on sample_id (the AEGIS id is NOT unique) and hands the confirmer rec["input"] — the exact object the validators ran on — so build_raw_spans re-derives the cited span idxs. ~49 bearing traces. - score_none_plus_validators.py: agent-only fold-in scorer (AEGIS gold has no step index). ORs the validator culprit_agent into the verifier=none agent set under the four verdict filters; reports agent Top-1/Hit/Exact-Set plus set-based micro/macro P/R/F1 (AutoJudge reference math, reused from verifier_ablation). Also reports the bearing-only subset (49/600) where the signal lives; F1 is the honest arbiter since the fold-in trades precision for recall. - verifier_ablation.py: EvidenceVerifier none/strict/soft ablation for AEGIS (LLM findings only), with the set-based P/R/F1. Result (bearing-only, confirmed filter vs none): Agent Hit 63.3->79.6, F1 micro 48.2->53.0, Recall 63.5->77.8; confirmer prunes 24 benign of 109 findings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply the Who&When/TraceElephant/AEGIS approach to TRAIL (GAIA). The judge findings were provided (trail_gemini_findings_v1/, 117 traces), so this only adds the confirmer + scorers. - gen_validators_confirm.py: attach the LLM confirmation layer to TRAIL's stored non_llm_validators. Reloads the raw trace from the GAIA dir by trace_id and hands the confirmer the raw dict so build_raw_spans (-> trail_to_spans) re-derives the native hex span_id idxs. ~47 bearing traces. - score_none_plus_validators.py: location fold-in scorer. TRAIL gold is a span-hash location (no agent -> location-only, the mirror of agent-only AEGIS). The crux is a namespace bridge: LLM judges cite numeric span positions while gold + validators use span-hashes, so judge idxs are mapped position->hash via trail_to_spans. Reports surface vs appointed locus, the four verdict filters, overall + bearing. - verifier_ablation.py: none/strict/soft location ablation (LLM findings only). Results (Step Hit; Top-1 is 0% throughout — judges never top-1 a gold span): - ablation none≈soft 16.8% > strict 12.4% (same shape as the other benches). - fold-in SURFACE adds nothing (raw validator idx never hits a gold span). - fold-in APPOINTED doubles recall: overall 16.8->30.1, bearing-only 34.8->67.4. The confirmer relocates each finding from its wrong surface span to the causal gold span -- appointing is the whole story on TRAIL (confirmed==all: only 7 benign of 488, so the verdict filter barely moves). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ortability
Consolidate the pipeline work so it runs and pulls cleanly on any machine.
- trail/calc_metrics_trail_findings.py: canonical LoCc (location accuracy) scorer
— reproduces the 18.3% exact baseline.
- trail/calc_metrics_trail_foldin.py: validator + LLM-confirmer fold-in on the
canonical LoCc metric (reuses resolve_location/IDX_SHIFT). Appointed locus lifts
exact LoCc 18.3->24.5% overall, 18.9->34.5% bearing.
- who_and_when/launch_findings_judges.py: shared-steps refactor — one coerced step
list feeds judges + validators + inline confirmer (no per-path re-coercion; runs
without .tolist()).
- who_and_when/{apply_llm_confirm,regen_validators_confirm,score_none_plus_validators}.py
+ reports: WW confirmer + fold-in + ablation.
- Machine-specific data paths made $ENV-overridable with a local fallback
(AEGIS_SRC, TRAIL_GAIA_DIR, TRAIL_ANNO_DIR) across the aegis/trail gen, trail
score, trail calc, and trail launcher — so no per-machine line edits (= no pull
conflicts).
- RUN_COSTS.md: per-run token/cost estimate (gemini-2.5-flash actual + gpt-4o),
validated to 1-4% against the two runs with recorded task_stats.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Report .md files are regenerable scoring outputs, not source — untrack the ones this branch added and ignore examples/**/reports/ so they don't get committed again. (The 4 pre-existing agentrx reports on main are left as-is.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every launch script now meters cost/latency like the who&when v9 runs: - metrics/base.py: LLMMetric.evaluate stashes the run's token usage on self.last_usage (handles pydantic_ai usage being a method or a property). - run_stats.py: extract_usage() (version-tolerant token pull) + aggregate_task_stats(). - who_and_when / trace_elephant / aegis / trail launchers: time each metric with perf_counter, read last_usage, and write a per-task `task_stats` block (wall_s, sum_metric_s, input/output/total_tokens, metrics_ok/failed) into each result file. Verified end-to-end: a real gemini-2.5-flash metric call captures in=1627 out=100. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Updated langfuse dependency version from 3.6.1 to 3.14.4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the deterministic-validator + LLM-confirmer fold-in and metrics across benchmarks, plus a WW launcher refactor. Reports are intentionally not tracked (see below).
What's here
gen_validators_confirm.py,score_none_plus_validators.py,verifier_ablation.py,score_swe_tool_attribution.py(SWE gold is a tool → scored in tool namespace).gen_validators_confirm.py(confirmer viasample_idtrace-match),score_none_plus_validators.py(agent-only, F1-led),verifier_ablation.py.calc_metrics_trail_findings.py(canonical LoCc = 18.3% exact baseline),calc_metrics_trail_foldin.py(fold-in on LoCc — appointed locus lifts exact 18.3→24.5% overall, 18.9→34.5% bearing),gen_validators_confirm.py,score_none_plus_validators.py,verifier_ablation.py..tolist()),apply_llm_confirm.py,regen_validators_confirm.py,score_none_plus_validators.py.src/maseval/validators/llm_confirm.py: two-knob confirmer (full reading view + windowed appointing).Portability (no per-machine pull conflicts)
Machine-specific data paths are
$ENV-overridable with a local fallback:AEGIS_SRC,TRAIL_GAIA_DIR,TRAIL_ANNO_DIR.Reports excluded
examples/**/reports/is gitignored; generated report.mdoutputs are untracked (regenerate via the scoring scripts). The 4 pre-existingagentrx/reports/*.mdon main are left untouched.🤖 Generated with Claude Code