Skip to content

fix: wrong dp-rank condition when enable pp #6374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 116 commits into
base: main
Choose a base branch
from
Open

fix: wrong dp-rank condition when enable pp #6374

wants to merge 116 commits into from

Conversation

liuqh16
Copy link

@liuqh16 liuqh16 commented Jul 30, 2025

📌 Checklist before creating the PR

  • I have created an issue for this PR for traceability
  • The title follows the standard format: [doc/gemini/tensor/...]: A concise description
  • I have added relevant tags if possible for us to better distinguish different PRs
  • I have installed pre-commit: pip install pre-commit && pre-commit install

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

📝 What does this PR do?

Fix the wrong dp-rank condition when enable pp:

  • If dp>1 and pp>1, execute rl_example will raise an ZeroDivisionError error because self.accum_count only accums when self.dp_rank==0 in grpo_consumer.py.

💥 Checklist before requesting a review

  • I have linked my PR to an issue (instruction)
  • My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
  • I have performed a self-review of my code
  • I have added thorough tests.
  • I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

  • 🌝 Yes, I do.
  • 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

ver217 and others added 30 commits February 21, 2025 15:24
@ver217
[chat] add distributed impl (hpcaitech#6210)
43c9b5f
@ver217
[feature] fit RL style generation (hpcaitech#6213)
de282dd 
* [feature] fit rl style generation

* [doc] add docstr

* [doc] add docstr
@TongLi3701
add reward related function
8e6c9a4
@TongLi3701
add simple grpo
ffd3878
@TongLi3701
update grpo
f736d74
@TongLi3701
polish
070907d
modify data loader
c15225b
grpo consumer
b96d690
update loss
678f5a9
update reward fn
d03cdea
update example
7f2ceac
update loader
812f4b7
add algo selection
0f566cc
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
ab5b6d8 
for more information, see https://pre-commit.ci
add save
0cc0c84
update select algo
0590f10
Merge branch 'grpo-latest' of github.com:hpcaitech/ColossalAI into gr…
22cc155 
…po-latest
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
eb6337f 
for more information, see https://pre-commit.ci
update grpo
9d9d516
update reward fn
754b16d
update reward
71a0181
fix reward score
abca66e
add response length
47d6493
detach
704866a
fix tp bug
131eece
fix consumer
afddfde
convert to 8 generation
4702d57
print results
45ac6c6
setup update
57b49da
@YeAnbang
fix transformers backend
bc0171d
YeAnbang and others added 26 commits June 9, 2025 09:37
@YeAnbang
fix bug, tested
3bed6ae
@YeAnbang
remove debug code
d0e12c5
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
9ca920c 
for more information, see https://pre-commit.ci
@YeAnbang
Merge pull request hpcaitech#6341 from hpcaitech/grpo-code
c308b42 
[feat] Support Code Generation RFT, Move Reward Calculation to Producer
@TongLi3701
move out evaluation func (hpcaitech#6343)
bb6f5d9 
Co-authored-by: Tong Li <[email protected]>
@YeAnbang
Manually schedule resources and support auto master address assigning
21d517d
@YeAnbang
modify readme
2559924
@YeAnbang
update readme
dc29c74
@YeAnbang
add ray timeout handling instruction
1330b57
@TongLi3701
fix pp memory issue (hpcaitech#6344)
8992def 
Co-authored-by: Tong Li <[email protected]>
@YeAnbang
Update README.md
2f02a28
@YeAnbang
Merge pull request hpcaitech#6316 from hpcaitech/grpo-support-multi-m…
ac06935 
…achine

[feat] Manually schedule resources and support auto master address assigning
@YeAnbang
fix num_update_per_episode
51b7abe
@YeAnbang
Merge pull request hpcaitech#6347 from hpcaitech/hotfix/fix_num_updat…
0e69b98 
…e_per_episode

[fix] fix num update per episode
@YeAnbang
optimize pp log_softmax OOM
30a6859
@YeAnbang
implement memory efficient logprob
e3d56cb
@YeAnbang
fix small bug
6b06430
@YeAnbang
Merge pull request hpcaitech#6348 from hpcaitech/grpo_optimization
dd49444 
[Feat] optimize pp log_softmax memory usage
@TongLi3701
add dp rank for multi-dp (hpcaitech#6351)
8880b83 
Co-authored-by: Tong Li <[email protected]>
@YeAnbang
[feat[ Support one-behind to reduce bubble time. Add profiling code (h…
b1f646c 
…pcaitech#6353)

* support n_behind, add profiling

* fix bugs

* fix visualization

* fix behind

* fix loop issue

* add profiling

* fix update

* update assert

* remove assert

---------

Co-authored-by: Tong Li <[email protected]>
@YeAnbang
fix code evaluation
a992da9
@YeAnbang
fix style
d850475
@YeAnbang
Merge pull request hpcaitech#6361 from hpcaitech/grpo-latest-fix-code…
f5c155a 
…-reward

[fix] Grpo latest fix code reward
@YeAnbang
add entropy (hpcaitech#6363)
4cf5ce2
@YeAnbang
hotfix entropy calculation (hpcaitech#6364)
57e9210
@liuqh16 @patrick-g-zhang
fix: wrong dp-rank condition when enable pp
d7b140d
@liuqh16 liuqh16 requested a review from a team as a code owner July 30, 2025 10:54
@TongLi3701 TongLi3701 changed the base branch from grpo-latest to main August 18, 2025 03:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@liuqh16 @ver217 @TongLi3701 @YeAnbang @duanjunwen