Add eval mode support for OpenCompass workflows

[dittops]

Summary

• add explicit eval modes to evaluation requests and datasets to distinguish generation vs perplexity runs
• propagate the selected mode into OpenCompass job scripts, enabling PPL flags, log-prob gateway options, and mode-specific output paths

Testing

• not run (not requested)

---

Codex Task

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[gemini-code-assist]

Summary of Changes

Hello @dittops, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the OpenCompass workflow by integrating explicit evaluation modes, specifically for generation and perplexity runs. It modifies the core evaluation request and dataset schemas to incorporate these modes and updates the job script generation logic to reflect the chosen mode. This change provides greater control and clarity over evaluation processes, allowing for tailored execution and output management based on whether a generation or perplexity evaluation is being performed.

Highlights

• Introduction of Evaluation Modes: Explicit EvalMode (Generation and Perplexity) has been introduced and added to EvaluationRequest and EvalDataset schemas, allowing for clear distinction between different evaluation types.
• Dynamic OpenCompass Job Script Generation: The OpenCompassTransformer now dynamically generates job scripts based on the selected eval_mode, enabling specific flags like --ppl and configuring log_probs gateway options as needed.
• Mode-Specific Output Paths: Output directories for OpenCompass runs are now dynamically named to include the eval_mode, ensuring better organization and traceability of evaluation results.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for distinct evaluation modes, specifically 'generation' (gen) and 'perplexity' (ppl), into the OpenCompass evaluation workflows. The changes are well-implemented through schema updates in evals/schema.py and corresponding logic adjustments in the OpencompassTransformer. The transformer now correctly propagates the evaluation mode to generate mode-specific output paths, command-line flags, and model configurations. My review includes one high-severity suggestion to refine the logic for determining the evaluation mode to ensure consistency when handling backward compatibility with legacy parameters, preventing potential inconsistencies in the evaluation setup.

[gemini-code-assist]

The current logic for determining if a PPL evaluation is requested can lead to inconsistencies. ppl_enabled can be true due to extra_args, but mode_flag and output_dir will not reflect this as they only depend on eval_mode. This can result in a PPL configuration being generated for a non-PPL run command, and results being saved in a misleadingly named directory.

To ensure consistency, the eval_mode should be the single source of truth. It should be determined first, taking into account backward compatibility with extra_args, and then all other variables (ppl_enabled, mode_flag, output_dir) should be derived from it.

Suggested change

	eval_mode = dataset.eval_mode or request.eval_mode
	ppl_enabled = eval_mode == EvalMode.PPL or bool(
	request.eval_model_info.extra_args.get("ppl", False)
	)
	eval_mode = dataset.eval_mode or request.eval_mode
	if dataset.eval_mode is None and request.eval_model_info.extra_args.get("ppl"):
	eval_mode = EvalMode.PPL
	ppl_enabled = eval_mode == EvalMode.PPL