Commit 53c564d
Fix step logic in DeepSpeedZeroOptimizer for calling ZenFlow
- Updated gradient tensor copying logic to accommodate ZenFlow's dimensionality and contiguity requirements.
- Added conditional handling for gradient updates based on the zenflow flag, ensuring compatibility with both standard and ZenFlow optimizers.
Signed-off-by: Tingfeng Lan <erc8gx@virginia.edu>
Co-authored-by: Yusen Wu <xrn4ub@virginia.edu>1 parent 06b15f3 commit 53c564d
1 file changed
+2
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1941 | 1941 | | |
1942 | 1942 | | |
1943 | 1943 | | |
| 1944 | + | |
| 1945 | + | |
1944 | 1946 | | |
1945 | 1947 | | |
1946 | 1948 | | |
| |||
0 commit comments