Confused with Multi-GPU in PL

[yipliu]

Hi,

I have read the doc here and here for validation_step_end.

My confusion is that we just need to set sync_dis=True in validation_step describe in here or we should do just like in here for validation_step_end.

[rohitgr7]

it says:

When running in distributed mode, ...

DP is not distributed training rather just data-parallel. So if you are using DP then go with here for validation_step_end else if using DDP or any other distributed strategy go with sync_dist=True.

Feel free to send a PR to update the docs, if you think they can be improved :)

[yipliu]

it says:

When running in distributed mode, ...

DP is not distributed training rather just data-parallel. So if you are using DP then go with here for validation_step_end else if using DDP or any other distributed strategy go with sync_dist=True.

Feel free to send a PR to update the docs, if you think they can be improved :)

Just to make sure.

If we just train the model using a strategy that splits data, we need to rewrite the training/validation/test_step_end function.

If we use the distributed strategy, we just need to sync_dict and we do not need to rewrite the training/validation/test_step_end function?

[rohitgr7]

yes and

we need to rewrite the training/validation/test_step_end function.

only if you are calculating some metrics.