Understanding epoch metrics: acc_train_epoch does not appear to be the average of acc_train_step

[cfhammill]

Hi lightning devs and users,

I’m using lightning to train some models for work and I’m having trouble understanding how the epoch level metrics are getting aggregated and computed. In my model the acc_train_step hits perfect accuracy and maintains that for 1000s of steps, where the acc_train_epoch stays < 0.7. From reading the documentation I would expect the acc_train_epoch to be the average acc_train_step for each step of the epoch, but then shouldn’t acc_train_epoch be 1 as well?

Can someone help me understand why these two graphs would be so different?

I’m using pytorch-lightning 1.6.3
python 3.9.10

Thanks!

extra details:
I’m training with parallel strategy dp on 4 gpus,
my training step looks like

def training_step(self, batch, batch_idx):
        x = batch["image"][tio.DATA]
        y = batch["label"]

        preds = self(x)
        y = y.view(y.shape[0], 1).float()
        loss = self.criterion(preds, y)
        acc = ((y > 0.5) == (preds > 0.5)) \
                   .type(torch.FloatTensor).mean()

        # perform logging
        self.log("train_loss", loss, on_step=True, on_epoch=True, prog_bar=True, logger=True, batch_size = x.shape[0])
        self.log("train_acc", acc, on_step=True, on_epoch=True, prog_bar=True, logger=True, batch_size = x.shape[0])

[rohitgr7]

it should be the same... normally it's just weighted average w.r.t the batch_size, but as you mentioned, it's 1 mostly, so epoch one must be 1 too. Is it possible for you to share a reproducible example to check this issue?

[cfhammill]

thanks @rohitgr7,nice to hear that my confusion is not misplaced. Unfortunately I think making a reprex would be challenging, I'm using an internal medical imaging dataset, so I'd need to make a synthetic data set and who knows if the issue would recapitulate there. I will continue attempting to figure it out on my end and if I'm able I'll cook up a reprex.

If it's any help I think the acc_train_step is correct, not the epoch result, the train loss is very small.

[rohitgr7]

can you just try logging 1.0.

self.log("train_acc", 1.0, on_step=True, on_epoch=True, prog_bar=True, logger=True, batch_size = x.shape[0])

just to see what might be going wrong here since it's the same as logging the actual metric. You can avoid training completely, too since acc is fake here.

[rohitgr7]

or maybe for some steps log something < 1.0 and then start logging. Just to replicate the behavior of the current graph.

acc = random.random() / 2 if self.global_step < 1000 else 1.0
self.log("train_acc", acc, on_step=True, on_epoch=True, prog_bar=True, logger=True, batch_size = x.shape[0])