Open
Conversation
… a backward_utils.py file
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
High-level Description
Added split backward pass support to Piper, which enabled the system to support a wider array of schedules (e.g., ZeroBubble).
Description of changes:
piperdirectory/project and renamed files accordingly by removing the "piper" prefix (if you prefer to keepsrcas the directory name, that's totally cool too)backward_weight()andbackward_input()functions toactor.py, which are the functions that implement the split backward passbackward_utilsfile containing helper functions for the split_backward_pass. These functions are essentially identically to the ones that PiPPy uses outside of a few minor tweaks.Taskclass inexec.pyto store aTaskTypeenum instead of booleans when determining what type of task type (f, b, u, i, w) each object representsvalidate_schedule()function inexec.pyto use the updatedTaskclass and check for correctness when backward_input/backward_weight tasks could be present.piper_exec()inexec.pyto include cases for handlingbackward_inputandbackward_weighttaskszb1pas an option to thetest_llama.py--scheduleargumentschedule_helpers.pyto use the updatedTaskclass