GSoC 2026 — Idea #2: Behavioral Evaluation Test Framework (POC for review)

[ancjainil]

Hi @gundermanc,

I'm Jainil Rana, and I'm applying for GSoC 2026 Idea #2: Behavioral Evaluation Test Framework for Gemini CLI.

I've been studying the existing eval infrastructure you've built — the evalTest harness, the ALWAYS_PASSES/USUALLY_PASSES classification, the nightly CI workflow, and your recent work on the evals README guidelines (#20629) and CI-gated eval runs (#20898). Your work on this foundation is exactly what drew me to this project.

A bit about my background: I currently work as a PR Writer and code quality evaluator on Anthropic's Code Human Preference pipeline (via Alignerr), where I perform multi-turn code reviews and score model outputs across axes like correctness, error handling, modularity, and production readiness. Previously, I spent 6 months at Outlier AI evaluating AI model outputs for coding tasks across Python, JavaScript, and SQL. I've also contributed to Prometheus (PR #17958 — Probe Listener feature) and Apache Kafka's open-source codebases.

I've built a proof-of-concept implementation of the full framework and would really appreciate your feedback on the approach:
POC Repository: https://github.com/ancjainil/GSoC_Gemini_CLI

The POC includes:

• A declarative scenario DSL that wraps evalTest with standardized fixtures, file/tool expectations, and custom assertions
• Multi-axis scoring engine (correctness, tool selection, efficiency, safety) with Wilson score confidence intervals
• Regression detection using two-tailed proportion tests against stored baselines
• HTML dashboard report generator for visualizing pass rates and regressions
• GitHub Actions workflow for nightly evals and PR-blocking CI gates
• Three example scenarios spanning debug, refactor, and code review categories

I'd love to hear your thoughts on:

1. Whether the declarative scenario DSL aligns with how you envision the eval infrastructure scaling
2. If the multi-axis scoring approach adds value beyond the current pass/fail model
3. Any concerns about the regression detection methodology or CI integration approach

I've also prepared a detailed proposal document that I'm happy to share if you're interested.

Thank you for your time, and for building the eval foundation that makes this project possible.

Best,
Jainil Rana
GitHub: @ancjainil
Email: jainilrana503@gmail.com