confusion about self.log() in test_step()

[yc1999]

Hi, I wrote following code in test_step() method:

    def test_step(self, batch, batch_idx):
        ...
        self.log("test_loss", loss, on_epoch=True, prog_bar=True)
        self.log("test_acc", acc, on_epoch=True, prog_bar=True)
        ...

According to the official document, Pytorch Lightning will cache every test_loss and test_acc value produced in test_step() and then do a mean reduction at the end of the test epoch.

Now, I set my number_of_test_batch=10000, this means the code will run test_step() 10000 times, and there will be a test_acc_lst whose len is 10000, and test_loss_lst whose len is 10000, and there will be sum(test_acc_lst)/10000 operation, is that right?

finally, when I check the tensorboard, I got following two pics:

I am very confused why there is a 1500(1.5K) step appeared in the picture, from the point of mine, it should be 10000 !!!
Can anyone solve this problem, thank you 😥

[carsondenison]

It looks like there's confusion between "steps", "epochs", and "batches". My hypothesis is that your code is running test_step() 10000 times, but that 10000 iterations is only 1500 epochs, and you have the flag on_epoch=True set.

Can you run some diagnostics? First, set the number of test batches to 1000 or even 100 instead of 10000. This will make things much faster while we iterate. Then: self.global_step will return the number of steps taken by the lightning module. You should log this in your training_step function with self.log('global step', self.global_step). You can also log self.log('on_epoch global step', self.global_step, on_epoch=True). This will tell us how many times test_step() is run, and how many epochs have been run.

Once you've run some tests and have a little more information, hopefully we can find the solution to your problem.

[tchaton]

Would you mind adding self.log(...., on_step=True, on_epoch=True). Right now, your metric is reduced across the entire epoch.