Skip to content

Commit 53c564d

Browse files
AntleraJoshWoo2003
andcommitted
Fix step logic in DeepSpeedZeroOptimizer for calling ZenFlow
- Updated gradient tensor copying logic to accommodate ZenFlow's dimensionality and contiguity requirements. - Added conditional handling for gradient updates based on the zenflow flag, ensuring compatibility with both standard and ZenFlow optimizers. Signed-off-by: Tingfeng Lan <erc8gx@virginia.edu> Co-authored-by: Yusen Wu <xrn4ub@virginia.edu>
1 parent 06b15f3 commit 53c564d

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

deepspeed/runtime/zero/stage_1_and_2.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1941,6 +1941,8 @@ def _optimizer_step(self, group_no):
19411941
elif self.torch_autocast_gradscaler:
19421942
self.torch_autocast_gradscaler.step(self.optimizer)
19431943
self.torch_autocast_gradscaler.update()
1944+
elif self.zenflow:
1945+
self.zenflow_cpu_optimizer_step(group_no)
19441946
else:
19451947
self.optimizer.step()
19461948

0 commit comments

Comments
 (0)