-
|
I have a question about using MPI. Specifically, the documentation says: "Note that the parallelism mode is data parallelism, so it is not expected to see the training time per batch decreases." When I run a job (with "--mpi-log=workers") on 2 GPUs, I get something like this:
How do I have to interpret this? As stated above, the training time stays the same with 1 or 2 GPUs. The batch, here 8100, also appears twice. Is something set up incorrectly in this case as I don't want to calculate the same batch twice or is this the expected behaviour? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
Two ranks are initialized using different random seeds, so their training data in each batch is different. |
Beta Was this translation helpful? Give feedback.
-
|
Ok thanks, but doesn't this mean that when I set e.g. numb_steps=10000 and use 2 GPUs then it trains it on 20000 batches? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it's the expected behavior. |
Beta Was this translation helpful? Give feedback.
-
|
I cannot set the seed myself, right? |
Beta Was this translation helpful? Give feedback.
Two ranks are initialized using different random seeds, so their training data in each batch is different.