Respect selected eval type for budeval runs

[dittops]

Summary

• read the requested eval type from the workflow data when triggering budeval evaluations and default to gen
• include the chosen eval_type in dataset items and validate datasets support the requested type

Testing

• Not run (not requested)

---

Codex Task

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[gemini-code-assist]

Summary of Changes

Hello @dittops, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the budeval evaluation triggering mechanism by introducing the ability to dynamically select the evaluation type based on workflow data. Previously, the system was hardcoded to use the 'gen' evaluation type. With this change, the system can now read a specified evaluation type, defaulting to 'gen' if none is provided. This improves the flexibility and accuracy of evaluations by ensuring they are performed using the correct, user-defined criteria, and includes robust error handling for unsupported evaluation types.

Highlights

• Dynamic Evaluation Type Selection: The system now reads the requested evaluation type from the workflow data when triggering budeval evaluations, allowing for more flexible and specific evaluation criteria.
• Default Evaluation Type: If no specific evaluation type is provided in the workflow data, the system gracefully defaults to using the 'gen' evaluation type.
• Dataset Validation and Inclusion: The pull request introduces validation to ensure that selected datasets support the requested evaluation type. The chosen evaluation type is also explicitly included in the dataset items used for evaluation.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates the evaluation triggering logic to respect the eval_type selected in the workflow, defaulting to 'gen' if not specified. It correctly adds validation to ensure that datasets support the requested evaluation type, raising an exception for mismatches. The changes also improve error handling by re-raising exceptions instead of swallowing them, and by failing fast on configuration issues, which enhances robustness. My feedback includes a couple of suggestions to simplify the code by removing redundant checks, which will improve readability and maintainability.

[gemini-code-assist]

The logic for retrieving requested_eval_type is a bit overly defensive. The _get_accumulated_step_data method is type-hinted to return a dict, and its implementation ensures it. The multiple isinstance checks can be simplified, making the code more concise and readable.

Suggested change

	workflow_data = await self._get_accumulated_step_data(workflow_id)
	step_five_data = workflow_data.get("step_5", {}) if isinstance(workflow_data, dict) else {}
	requested_eval_type_raw = step_five_data.get("eval_type") if isinstance(step_five_data, dict) else None
	requested_eval_type = str(requested_eval_type_raw).lower() if requested_eval_type_raw else "gen"
	workflow_data = await self._get_accumulated_step_data(workflow_id)
	step_five_data = workflow_data.get("step_5", {})
	requested_eval_type_raw = step_five_data.get("eval_type")
	requested_eval_type = str(requested_eval_type_raw).lower() if requested_eval_type_raw else "gen"

[gemini-code-assist]

The if eval_types check here is redundant. This code is inside a block that has already confirmed eval_types is a truthy dictionary (if eval_types and isinstance(eval_types, dict):). Removing this unnecessary conditional will make the code slightly cleaner.

Suggested change

	available_types = ", ".join(eval_types.keys()) if eval_types else "none"
	available_types = ", ".join(eval_types.keys())