resample_abs_pos_embed does not apply when checkpoint_path is used

[Luciennnnnnn]

I'm creating a ViT model with following code:

model = timm.create_model('vit_base_patch16_224', pretrained=False, img_size=64,
                                  checkpoint_path=local_checkpoint_path,
                                  global_pool='avg')

Where I use a img_size 64.
I expect resample_abs_pos_embed will also applies to this use case like loading pretrained model from timm.

[rwightman]

@LuoXin-s you can use

timm.models.load_checkpoint(model, 'checkpoint.pth', filter_fn=timm.models.vision_transformer.checkpoint_filter_fn)

if no other adaptation is needed (ie # classes, input channel change, etc), or

timm.create_model(
  'vit_base_patch16_224', pretrained=True, img_size=64, global_pool='avg',
  pretrained_cfg_overlay=dict(file='checkpoint_path'),
)

[Luciennnnnnn]

Hi @rwightman , this solve my problem. Nevertheless, I have another question, can we provide state dict to timm.create_model or timm.models.load_state_dict directly (other than provide a path)? Also, it should apply filter_fn.
I'm doing a self-supervised pretraining, I want to load partial pretrained weights (can be accessed from net.state_dict()) to a timm ViT for validation, which will add classification head easily.

Here is the desired behavior:

timm.models.load_state_dict(model, pretrained.state_dict(), filter_fn=timm.models.vision_transformer.checkpoint_filter_fn)

or

timm.create_model(
  'vit_base_patch16_224', pretrained=True, img_size=64, global_pool='avg',
  pretrained_cfg_overlay=dict(state_dict=pretrained.state_dict()),
)

[rwightman]

@LuoXin-s woops, started editing inline with your ans but that's confusing so reverted back, there's no way to pass a state dict in directly and have it do the upsample/filter, you'd have to manually run the filter fn your state dict (it's just model and state dict as input and state dict as output), adjust the state dict in any other way you need, and then load it into the model