Is there any function to gather generated data on different nodes? #10314
Unanswered
ElderWanng
asked this question in
DDP / multi-GPU / multi-node
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I'm writing a project about a generative model.
I use
self.gather()
to make every process have all generated tensor. then decode such a big tensor to get the result.It works the first time, but later I found as the number of machines and GPUs increased, the proportion of time taken up to sync results by using tensor become larger.
Actually, the final result is just a batch of predicted strings with their id in the dataset. I want to know if there is any good API to simply gather all predicted objects ( strings, lists ) into one process?
Beta Was this translation helpful? Give feedback.
All reactions