[Feature] Return table for service and fix analysis bug

[Yunnglin]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the evaluation and performance services by integrating formatted result tables directly into their API responses, providing a more structured and easily digestible summary of metrics. It also addresses a bug in the LLM judge analysis generation, ensuring that disclaimers are only appended to successfully generated analyses, thereby improving the robustness of the reporting mechanism.

Highlights

• Service Table Output: The evaluation and performance services now return a formatted Markdown table of results in their JSON responses, improving readability for users.
• Analysis Bug Fix: Corrected an issue in the LLM judge analysis generation where a disclaimer was incorrectly added even when the analysis failed.
• LLM Judge Configuration Refinement: Updated the default generation_config in the LLMJudge constructor to an empty dictionary, allowing for more flexible and dynamic configuration.
• Model ID Parameter Alignment: Changed the parameter name from model_id to model when initializing LLMJudge in the report generation, aligning with the TaskConfig structure.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a feature to return a formatted markdown table for evaluation and performance results in the service layer, which improves the usability of the API. It also includes a bug fix for handling errors during the generation of analysis reports. While these are valuable additions, I've identified two high-severity issues that could impact the correctness and stability of the evaluation process. One change may lead to non-deterministic judge model outputs, and another could cause runtime errors due to a potential type mismatch. Please see the detailed comments for suggestions on how to address these.

[Copilot]

Pull request overview

This PR enhances the EvalScope Flask service responses by adding a human-readable Markdown table for both evaluation and performance endpoints, and fixes a bug in analysis generation around which model identifier is used.

Changes:

• Add a table field to /api/v1/eval/invoke responses by building a Markdown pipe-table from JSON reports.
• Add a table field to /api/v1/perf/invoke responses by building a Markdown pipe-table from perf benchmark results.
• Fix analysis generation to skip storing LLM error strings and switch model identifier usage for LLMJudge.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
evalscope/service/blueprints/perf.py	Builds a Markdown pipe-table from perf results and returns it in the service response.
evalscope/service/blueprints/eval.py	Builds a Markdown pipe-table from report JSONs and returns it in the service response.
evalscope/report/report.py	Adjusts LLMJudge initialization for analysis and skips analysis on [ERROR] responses.
evalscope/metrics/llm_judge.py	Changes default generation_config behavior for judge calls.