Logging sync_dist Documentation Clarification

[Unturned3]

📚 Documentation

If we do self.log('loss', loss, sync_dist=True) in a LightningModule, is the loss averaged or summed across all the GPUs? From my experiments it seems to be averaged, but it would be nice if the documentation can confirm this explicitly. Currently it says "reduces the metric across devices" but does not mention anything about the reduction operation.

Edit: Upon further investigation, the default reduction operation used by self.log seems to be trainer.strategy.reduce (module.py, line 534). At least for DDPStrategy, this indeed defaults to a mean reduction.

cc @lantiga @Borda

[bhimrazy]

Yeah, seems like mean is the default for all strategies.