Question about logging metric object with on_step=True

[ashleve]

I have a hard time understanding how logging value through metric object works when using on_step=True.

I have these metrics in my lightning module:

# for calculating and averaging accuracy across batches
self.train_acc = Accuracy()

# for averaging loss across batches
self.train_loss = MeanMetric()

I'm logging metrics on step like this:

def training_step(self, batch: Any, batch_idx: int):
    loss, preds, targets = self.step(batch)

    # update metrics
    self.train_loss(loss)
    self.train_acc(preds, targets)
    
    self.log("train/loss", self.train_loss, on_step=True, on_epoch=False, prog_bar=True)
    self.log("train/acc", self.train_acc, on_step=True, on_epoch=False, prog_bar=True)

Question: I'm seeing metric values on the progress bar on each step. Are these values actually accuracy and loss only from current step, or maybe these values are average loss and average accuracy from all steps so far in current epoch?

[akihironitta]

Question: I'm seeing metric values on the progress bar on each step. Are these values actually accuracy and loss only from current step,

Yes, the value of train/loss=... shown in the progress bar is calculated based on each step. However, note that the value of loss=... (shown by default) at the very left of your progress bar (not train/loss=...) is smoothened across 20 steps. (Related to this RFC #9372)

[yipliu]

Hi

How about the loss used for gradient accumulation?

In the doc, we just need to set the value of accumulate_grad_batches in Trainer.

However, in the standard PyTorch code, we need to get the normalized loss for loss.backward().

So, in PyTorch lightning

# self.criterion = torch.nn.CrossEntropyLoss()
def training_step(self, batch):
    loss, preds, targets = self.step(batch)
    self.log("train/loss", self.train_loss, on_step=False, on_epoch=True, prog_bar=True)
    return {"loss": loss}

The returned loss is not normalized. If we want to do gradient accumulation, do we need to normalize the loss manually? Or will it be handled by PL automatically?

WOW. I find code here, PL does it for us!