Add RunLedger replay gate for agent regressions

[ZackMitchell910]

Summary

• add a replay-only RunLedger eval suite (suite/case/schema/cassette + stub agent)
• add a baseline file for regression gating
• add a GitHub Actions workflow using runledger/Runledger@v0.1
• add a small README note + ignore runledger_out/

How to run locally

runledger run evals/runledger --mode replay --baseline baselines/runledger-demo.json

Notes

• no external calls; replay-only cassette
• feel free to remove the suite/workflow if it is not desired
• GitHub Actions note: workflows from first-time contributors/forks may require a maintainer to click “Approve and run” before checks will execute.

[gemini-code-assist]

Summary of Changes

Hello @ZackMitchell910, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates a new RunLedger-based CI gate to ensure the stability and prevent regressions in tool-using agents. By setting up a dedicated replay-only evaluation suite, defining a baseline for expected behavior, and automating its execution via GitHub Actions, the changes aim to provide deterministic checks against schema, tool usage, and budget deviations. This enhancement helps maintain the integrity of agent functionality and provides clear feedback on any unintended changes.

Highlights

• RunLedger Evaluation Suite: A new replay-only RunLedger evaluation suite has been introduced, comprising a suite definition, a test case, a schema, a cassette, and a stub agent.
• Regression Gating Baseline: A baseline file (baselines/runledger-demo.json) has been added to enable regression gating for agent performance and behavior.
• GitHub Actions Integration: A GitHub Actions workflow has been added to automate the execution of the RunLedger evaluation using runledger/Runledger@v0.1.
• Documentation Update: The README.md has been updated with a new section detailing the RunLedger CI gate and instructions for local execution.
• Git Ignore Configuration: The .gitignore file has been updated to exclude the runledger_out/ directory, which will contain RunLedger's output.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/runledger.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a RunLedger replay gate for agent regressions, which is a great addition for ensuring stability. The changes include a new evaluation suite, a baseline file for regression gating, and a GitHub Actions workflow. My review focuses on ensuring the new testing infrastructure is portable and maintainable. I've identified a critical issue in the baseline file where hardcoded absolute paths will prevent it from working in CI or for other developers. I've also suggested a minor refactoring in the new agent script to improve its structure and adhere to common Python best practices. Overall, this is a valuable contribution once the portability issue is addressed.

[gemini-code-assist]

This baseline file contains hardcoded absolute local paths which will prevent it from being used in a CI environment or by other developers. Specifically:

• cases[0].replay.cassette_path (line 71)
• suite.agent_command[1] (line 105)
• suite.suite_path (line 110)

These paths are specific to a single user's machine (/mnt/c/Users/snowb/...). To ensure portability and allow the CI gate to function correctly, these paths should be relative to the project root, or the process that generates this baseline file should be configured to not include machine-specific paths.

[gemini-code-assist]

For better code structure and maintainability, it's a good practice to encapsulate the main logic of a script in a function (e.g., main) and call it under an if __name__ == '__main__': guard. This makes the code more reusable and easier to test, should you decide to add unit tests later.

Suggested change

	for line in sys.stdin:
	line = line.strip()
	if not line:
	continue
	msg = json.loads(line)
	if msg.get("type") == "task_start":
	ticket = msg.get("input", {}).get("ticket", "")
	send({"type": "tool_call", "name": "search_docs", "call_id": "c1", "args": {"q": ticket}})
	elif msg.get("type") == "tool_result":
	send({"type": "final_output", "output": {"category": "account", "reply": "Reset password instructions sent."}})
	break
	def main():
	for line in sys.stdin:
	line = line.strip()
	if not line:
	continue
	msg = json.loads(line)
	if msg.get("type") == "task_start":
	ticket = msg.get("input", {}).get("ticket", "")
	send({"type": "tool_call", "name": "search_docs", "call_id": "c1", "args": {"q": ticket}})
	elif msg.get("type") == "tool_result":
	send({"type": "final_output", "output": {"category": "account", "reply": "Reset password instructions sent."}})
	break
	if __name__ == "__main__":
	main()

[ZackMitchell910]

Thanks for taking a look! This PR adds a replay-only RunLedger gate. The workflow runs are currently waiting on fork approval (action_required). If you are open to it, please approve/authorize the workflow run so CI can complete. Happy to adjust anything.