Skip to content

self.manual_backward() vs. loss.backward() when optimizing manually #11318

Discussion options

You must be logged in to vote

hey @MGheini

It's not just precision but a common hook to support all other strategies like deepspeed/ddp and certain hooks like on_after_backward are called too. So manual_backward is suggested to make sure no-code change is required for eg in case any of the strategies is updated by the user.

Replies: 1 comment 1 reply

Comment options

You must be logged in to vote
1 reply
@MGheini
Comment options

Answer selected by MGheini
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment