GSoC 2026: Behavioral Eval Framework - Taxonomy, Coverage, and Cross-Agent Benchmarking

[parinzee]

Hi, I'm Parin a student at CMU studying AI. I'm interested in the Behavioral Evaluation Test Framework GSoC project.

I've been reading through the existing evals/ infrastructure and the README. The evalTest + policy system and the /fix-behavioral-eval self-healing loop are really well-designed. I have a few questions about the GSoC project's intended scope:

• Coverage gaps: The existing evals seem to focus on memory and plan mode. Is the primary GSoC goal to expand coverage to other tools (shell, file ops, web search, codebase investigator), or also to add more complex multi-step workflow evals?
• Taxonomy: I noticed evals are currently organized per-feature. Would there be interest in a more systematic task taxonomy (e.g., categorized by reasoning depth, context scope, tool chain length) to make coverage gaps visible and guide future eval development?
• Cross-agent comparison: I also saw Setup evals using public leaderboard harnesses (for SWE-Bench and Terminal-Bench) #10898 (SWE-Bench/Terminal-Bench harness) Is there still appetite for making the eval scenarios agent-agnostic so other tools could be benchmarked against the same suite? I do think maybe expanding this to an agent-agnostic framework would be very interesting as well.

For context, my research involves designing cross-lingual evaluation benchmarks for multimodal LLMs, I've published on evaluation methodology as well so I'm pretty familiar with the entire benchmarking / evaluating LLMs. I'm particularly interested in adding long tail tasks so we could also measure behavior changes like "How well does the agent navigate larger codebases?" or other more complex scenarios.

I'm planning to submit a PR adding behavioral evals for currently-untested tools as a starting contribution. Happy to align with whatever areas are highest priority!

Best Regards,
Parinthapat Pengpun

cc: @gundermanc

[aniruddhaadak80]

From my point of view, the taxonomy question is one of the most valuable parts of this proposal because it changes evals from a pile of scenarios into something maintainers can actually reason about over time. I would still keep the first expansion tied to real workflow categories like debugging, review, and multi step editing before pushing too far into cross agent benchmarking. Once the suite describes the product well, the cross agent comparison story becomes much more credible.