Usage of Trainer.test

[Keredu]

Discussed in #10218

^{Originally posted by Keredu October 28, 2021}
Hi,

I would like to ask about how to use Trainer.test correctly. In this google colab link, I have developed a minimal example using MNIST Dataset where:

1. Train a model (LightiningModule), with a callback that stores both best and last ckpt as best.ckpt and last.ckpt, respectively.
2. Immediately after the training process, run trainer.test(model=model, datamodule=datamodule), which returns the same metrics that trainer.test(model=model, datamodule=datamodule, ckpt_path='.best.ckpt') and trainer.test(model=model, datamodule=datamodule, ckpt_path='best') , as expected.
3. Use a classical for loop to evaluate my model and it gives me different results:
   3.1. Evaluate using the model trained some chunks before: I get the same result that using trainer.test. I expected to obtain the results of the last epoch instead.
   3.2 Evaluate using model.resume_from_checkpoint('best.ckpt'): I get a completely different result than in the previous approaches.
   3.3 Evaluate using model.resume_from_checkpoint('last.ckpt): I get the same result than 3.1.

Thanks in advance for your time.

Trainer.test used:

trainer.test(model=model, datamodule=datamodule)
DATALOADER:0 TEST RESULTS
{'plt/test_accuracy': 0.8510000109672546}

trainer.test(model=model, datamodule=datamodule, ckpt_path='./best.ckpt')
DATALOADER:0 TEST RESULTS
{'plt/test_accuracy': 0.8510000109672546}

trainer.test(model=model, datamodule=datamodule, ckpt_path='best')
DATALOADER:0 TEST RESULTS
{'plt/test_accuracy': 0.8510000109672546}

Loop used to evaluate the model:

print('No loading weights.')
preds_tensor = None
targets_tensor = None
with torch.no_grad():
    for batch in datamodule.test_dataloader():
        samples, targets = batch
        preds = model(samples)
        _, preds = torch.max(preds, 1)
        if preds_tensor is None:
            preds_tensor = preds
            targets_tensor = targets
        else:
            preds_tensor = torch.cat([preds_tensor, preds])
            targets_tensor = torch.cat([targets_tensor, targets])
nn_module_acc = Accuracy()
print('Acc nn_module:', nn_module_acc(preds_tensor, targets_tensor))
print('nn_acc compute:', nn_module_acc.compute())

print(80 * '=')

print('Loading best weights.')
model = model.load_from_checkpoint('./best.ckpt')
preds_tensor = None
targets_tensor = None
with torch.no_grad():
    for batch in datamodule.test_dataloader():
        samples, targets = batch
        preds = model(samples)
        _, preds = torch.max(preds, 1)
        if preds_tensor is None:
            preds_tensor = preds
            targets_tensor = targets
        else:
            preds_tensor = torch.cat([preds_tensor, preds])
            targets_tensor = torch.cat([targets_tensor, targets])
nn_module_acc = Accuracy()
print('Acc nn_module:', nn_module_acc(preds_tensor, targets_tensor))
print('nn_acc compute:', nn_module_acc.compute())

print(80 * '=')

print('Loading last weights.')
model = model.load_from_checkpoint('./last.ckpt')
preds_tensor = None
targets_tensor = None
with torch.no_grad():
    for batch in datamodule.test_dataloader():
        samples, targets = batch
        preds = model(samples)
        _, preds = torch.max(preds, 1)
        if preds_tensor is None:
            preds_tensor = preds
            targets_tensor = targets
        else:
            preds_tensor = torch.cat([preds_tensor, preds])
            targets_tensor = torch.cat([targets_tensor, targets])
nn_module_acc = Accuracy()
print('Acc nn_module:', nn_module_acc(preds_tensor, targets_tensor))
print('nn_acc compute:', nn_module_acc.compute())

and returns:

No loading weights.
Acc nn_module: tensor(0.8510)
nn_acc compute: tensor(0.8510)
================================================================================
Loading best weights.
Acc nn_module: tensor(0.8369)
nn_acc compute: tensor(0.8369)
================================================================================
Loading last weights.
Acc nn_module: tensor(0.8510)
nn_acc compute: tensor(0.8510)

[rohitgr7]

hey, I think this was resolved over slack right? is it the same one?

[Keredu]

Yes, it's the same. Thanks for your help.

[Abermal]

For some reason, my test method doesn't print out the metric.
Here is my test step:

    def test_step(self, batch, batch_idx):
        x, y, _ = batch
        preds = self.forward(x)
        test_losses = self.valid_loss(preds, y)
        focal = test_losses["focal"] * self.focal_strength
        dice_loss = test_losses["dice_loss"] * (1 - self.focal_strength)
        test_loss = focal + dice_loss
        self.log('test_loss', test_loss, prog_bar=True)
        self.valid_metric(preds, y)

        return {
            'test_loss': test_loss.sum(),
            'test_mean_dist': self.valid_metric
        }

Meanwhile, validate works, but the labels are strange: