How to log with accumulate_grad_batches

[FrancescoSaverioZuppichini]

Hi Guys,

Since it is not implemented, could you please explain to me how I properly log the loss with accumulate_grad_batches > 1?

Thanks!

Francesco

[tchaton]

Hey @FrancescoSaverioZuppichini,

Right now, Lightning logs the latest value of the accumulate_grad_batches as the mean is not necessarily the right way to reduce a metric.
I would advice implementing your loss, metrics using TorchMetrics if you want to properly log for accumulate_grad_batches

class LossTensor(Metric):

    def __init__(self):
        self.add_states("loss", torch.tensor(0))
        self.add_states("counter", torch.tensor(0))

    def update(self, loss):
        self.loss += loss
        self.counter += 1

    def compute(self):
        return self.loss / self.counter


class Model(LightningModule):

    def __init___(self):
        ...
        self.loss_metric = LossTensor()


    def training_step(self, batch, batch_idx):
        loss = 
        self.loss_metric(loss) # accumulate

        if not self.trainer.fit_loop.should_accumulate()
            self.log("loss", self.loss_metric.compute())
            self.loss_metric.reset() # reset 

Best,
T.C

[FrancescoSaverioZuppichini]

Hi @tchaton,

For you for your reply :)

Sorry but I didn't quite get your answer

Right now, Lightning logs the latest value of the accumulate_grad_batches as the mean is not necessarily the right way to reduce a metric.

You mean that currently the loss logged is the mean from all the accumulated batches? If yes I don't need to use torchmetrics. If no, why it is not the default behaviour?

Thank you again!

Cheers,

Francesco

[tchaton]

Dear @FrancescoSaverioZuppichini,

I meant Lightning will log the latest logged value during accumulation phase. So if you want the mean, you will have to use TorchMetrics and log the mean on not self.trainer.fit_loop.should_accumulate().

[johan-sightic]

Hi @tchaton, is it still the case that I need to implement my own metric with TorchMetrics or has this been simplified?

Thanks!