Synchronize train logging

[yc1999]

as what is said in the above picture, validation and test log can use sync_dist=True.
I wonder whether there is a solution for training to synchoronize? like the following code, I run in 8 gpus, I want 8 gpus’s train_loss and train_acc to be averaged :

  def training_step(self, batch, batch_idx):

      inputs = self.train_inputs(batch)
      loss, logits = self(**inputs)

      mask = (batch['labels'] != 5).long()
      ntotal = mask.sum()
      ncorrect = ((logits.argmax(dim=-1) == batch['labels']).long() *
                  mask).sum()
      acc = ncorrect / ntotal

      self.log('train_loss', loss, on_step=True, prog_bar=True)
      self.log("train_acc", acc, on_step=True, prog_bar=True)

      return loss

[rohitgr7]

I believe this applies for training as well. self.log(..., sync_dist=True) should work fine.