[grpo] support GSPO-token & add GSPO script

[hjh0119]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

Write the detail information belongs to this PR.

Experiment results

Paste your experiment result here(if needed).

[gemini-code-assist]

Summary of Changes

Hello @hjh0119, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant enhancement to the Group Sequence Policy Optimization (GRPO) framework by adding support for the GSPO-token importance sampling method. This new method aims to improve the stability and effectiveness of importance weight calculation during reinforcement learning from human feedback (RLHF) training. The changes include core implementation, comprehensive documentation updates, and a new example script to guide users on its application.

Highlights

• New Feature: GSPO-token Support: Implemented the GSPO-token importance sampling method within swift/trainers/rlhf_trainer/grpo_trainer.py. This new method provides a hybrid approach for calculating importance weights by combining sequence-level and token-level considerations, aiming for more stable gradient estimation in GRPO training.
• Updated Documentation: Both Chinese (docs/source/Instruction/GRPO/AdvancedResearch/GSPO.md) and English (docs/source_en/Instruction/GRPO/AdvancedResearch/GSPO.md) documentation have been updated. They now include detailed explanations of the GSPO-token method, its mathematical formulation, theoretical background, and pseudo-code implementation, alongside the existing GRPO and GSPO explanations. The --importance_sampling_level parameter options have also been expanded to include sequence_token.
• New Example Training Script: A new shell script (examples/train/grpo/internal/gspo.sh) has been added. This script serves as a practical example for configuring and running GRPO training, demonstrating how to utilize the newly supported importance_sampling_level options, including sequence_token.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for Group Sequence Policy Optimization (GSPO) and GSPO-token, extending the existing GRPO implementation. The changes include documentation updates, a new training script, and core logic implementation in the GRPOTrainer. A critical issue in the new gspo.sh script will prevent it from running, and a high-severity issue exists where the script doesn't configure the training to use GSPO. A minor improvement to the Chinese documentation is also suggested for consistency.