String “best” at argument “ckpt_path” for test method of Trainer class

[dr-costas]

Hi,

The test method of the Trainer class, has the input argument ckpt_path. According to the docs:

ckpt_path (Optional[str]) – Either best or path to the checkpoint you wish to test. If None and the model instance was passed, use the current weights. Otherwise, the best model from the previous trainer.fit call will be loaded.

Also, in the documentation of PyTorch Lightning for the test set, using Trainer, there is the following:

# run full training
trainer.fit(model)

# (1) load the best checkpoint automatically (lightning tracks this for you)
trainer.test(ckpt_path="best")

My question is, according to what the "best" checkpoint is decided? That is, is the "best" decided on maximising or minimising some value? What would be that value? Can someone configure the policy (i.e. minimising or maximising) and the value? How one should use this "best" string?

Links for reference:

1. Test set documentation
2. Test method of Trainer class documentation

P.S. Please note that I'm not referring to using the ModelChekpoint callback, but explicitly to the above, which seems that the ModelCheckpoint callback is not used.

[rohitgr7]

it actually relies on ModelCheckpoint callback only https://github.com/PyTorchLightning/pytorch-lightning/blob/b3e9dff32d842431b067b0ab83e508ffe3262968/pytorch_lightning/trainer/trainer.py#L1268-L1280
so if there is no ModelCheckpoint callback it will raise an error

[dr-costas]

I see! That makes total sense, but it was never mentioned in the docs! Thank you for digging that up!

[dr-costas]

@rohitgr7

One more question. Would that work when using EarlyStopping callback?

[rohitgr7]

EarlyStopping has nothing to do with checkpoints so it won't. You need to have ModelCheckpoint configured for this.

[dr-costas]

Cool! Thnx!