Trainer test with ckpt_path not work as expected

[tshu-w]

I wander If I misunderstanding doc. Trainer test with ckpt_path doesn't load ckpt. model.load_from_checkpoint works.

In [20]: model = MMTSMatcher(model_name="bert-base-uncased")
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

In [21]: trainer.test(model, data, ckpt_path="tests/mmtsmatcher_wdcdatamodule/shoes_small_false_32_0808-182020/checkpoints/epoch=05-valid_f1=81.22%.ckpt")
The following callbacks returned in `LightningModule.configure_callbacks` will override existing callbacks passed to Trainer: ModelCheckpoint
/data/wts/.local/share/anaconda3/envs/multimodalER/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:423: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.
  rank_zero_deprecation(
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5]
Testing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 69/69 [01:27<00:00,  1.04s/it]--------------------------------------------------------------------------------
DATALOADER:0 TEST RESULTS
{'test_f1': 0.0,
 'test_loss': 0.6186538934707642,
 'test_prc': 0.0,
 'test_rec': 0.0}
--------------------------------------------------------------------------------
Testing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 69/69 [01:27<00:00,  1.27s/it]
/data/wts/.local/share/anaconda3/envs/multimodalER/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:423: LightningDeprecationWarning: DataModule.teardown has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.teardown.
  rank_zero_deprecation(
Out[21]:
[{'test_f1': 0.0,
  'test_prc': 0.0,
  'test_rec': 0.0,
  'test_loss': 0.6186538934707642}]

In [22]: model = MMTSMatcher.load_from_checkpoint(checkpoint_path="tests/mmtsmatcher_wdcdatamodule/shoes_small_false_32_0808-182020/checkpoints/epoch=05-valid_f1=81.22%.ckpt", hparams_file="tests/mmtsmatcher_wdcd
    ...: atamodule/shoes_small_false_32_0808-182020/hparams.yaml")
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

In [23]: trainer.test(model, data)
The following callbacks returned in `LightningModule.configure_callbacks` will override existing callbacks passed to Trainer: EarlyStopping, ModelCheckpoint
/data/wts/.local/share/anaconda3/envs/multimodalER/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:423: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.
  rank_zero_deprecation(
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5]
Testing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 69/69 [01:27<00:00,  1.21it/s]--------------------------------------------------------------------------------
DATALOADER:0 TEST RESULTS
{'test_f1': 0.670258641242981,
 'test_loss': 0.284279465675354,
 'test_prc': 0.5298126339912415,
 'test_rec': 0.9120234847068787}
--------------------------------------------------------------------------------
Testing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 69/69 [01:27<00:00,  1.27s/it]
/data/wts/.local/share/anaconda3/envs/multimodalER/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:423: LightningDeprecationWarning: DataModule.teardown has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.teardown.
  rank_zero_deprecation(
Out[23]:
[{'test_f1': 0.670258641242981,
  'test_prc': 0.5298126339912415,
  'test_rec': 0.9120234847068787,
  'test_loss': 0.284279465675354}]

[tchaton]

Dear @tshu-w,

Mind opening an issue with a reproducible script using the BoringModel.

Best,
T.C

[tshu-w]

Sure, when I'm free recently

[tshu-w]

I thought this issue is not exist in 1.5.2