feat: Update LLMGrader and schema

[jc200808]

OpenJudge Version

[The version of OpenJudge you are working on, e.g. import openjudge; print(openjudge.__version__)]

Description

[Please describe the background, purpose, changes made, and how to test this PR]

Checklist

Please check the following items before code is ready to be reviewed.

• Code has been formatted with pre-commit run --all-files command
• All tests are passing
• Docstrings are in Google style
• Related documentation has been updated (e.g. links, examples, etc.)
• Code is ready for review

[gemini-code-assist]

Summary of Changes

Hello @jc200808, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines the LLMGrader class to properly handle and persist model configurations during serialization. Concurrently, it strengthens the data integrity of grader results by making key fields explicitly required in the Pydantic schemas and implementing robust validation logic for rank-based grading outputs. These changes enhance the reliability and predictability of the grading system.

Highlights

• LLMGrader Model Configuration Persistence: The LLMGrader class now stores and serializes its model configuration when the model is provided as a dictionary, ensuring that the original configuration is preserved during object serialization.
• Improved Error Handling in LLMGrader: The initialization of LLMGrader has been made more robust by replacing an assert statement with a RuntimeError for missing essential attributes (language, template, or model), providing clearer error messages.
• Schema Field Requirement Enforcement: Several fields across various grader result schemas (GraderResult, GraderScore, GraderScoreCallback, GraderError, GraderRank, GraderRankCallback) are now explicitly required by removing default=... from their Field definitions.
• Enhanced Rank Validation: A comprehensive field_validator has been added to GraderRank and GraderRankCallback to ensure that rank lists are valid permutations of positive integers, preventing malformed or invalid rank data.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates the LLMGrader and its associated Pydantic schemas. The changes in LLMGrader are solid, improving serialization by storing the model configuration and making initialization checks more robust by replacing assert with explicit exceptions. In schema.py, several fields are now correctly marked as required, and a new validator for rank permutations has been added. My review focuses on improving the new validator's implementation by addressing a significant code duplication issue. Overall, these are good improvements to the grader's reliability and maintainability.