Idea #2 (Behavioral Evaluation): Structuring the Benchmark Suite & CI Integration

[Shreenath-14]

Hi @christian Gunderman and team! 👋

I’ve set up the CLI locally (v0.35.0-nightly) and am drafting a proposal for Idea 2 (Behavioral Evaluation Test Framework) from GSoC project idea list.

As a developer who relies heavily on LLMs for code generation, I am very familiar with the edge cases where agents fail (e.g., losing context in large refactors or hallucinating tool calls). I want to build a robust safety net for the CLI.

My architectural question regarding the test harness:
For executing the 50+ benchmark scenarios automatically, is the vision to run these evaluations using the mock file system (test-utils), or do you prefer spinning up isolated Docker sandbox environments for each test to evaluate the agent's actual shell execution and network interactions?

Also, for measuring the "success rate metrics", should the framework rely on deterministic assertions (e.g., checking if a specific string exists in the output file) or are you open to an "LLM-as-a-Judge" approach where another Gemini instance evaluates the quality of the generated code?

I'm currently structuring the categories for the benchmark suite (Debugging, Scaffold, Refactor, Tool Use) and want to align the execution environment with your CI/CD goals.

Looking forward to your thoughts!