Feature Extraction

[strugoeli]

Hi,
I want to use Efficientnet and Regnety for feature extraction and use the features for image similarity.
I want to verify that I'm doing it right:

Efficientnet:
x = model.forward_features(img)
x = model.global_pool(x)

Regnety:
x = model.forward_features(img)
x = model.head.global_pool(x)

Thank you in advance.

[rwightman]

@strugoeli there's actually some relevant discussion in this issue (feature request) #1141 .... but the creator of that issue was looking for both the embeddings and predictions.

If you just want the embeddings (the activations right before the classifier) that are shape (batch_size, num_features) the best way is

model = timm.create_model('resnet50', num_classes=0)`  # create model with no classifier (nn.Identity in place), but pooling still present

or

model = timm.create_model('resnet50')
model.reset_classifier(num_classes=0)  # remove classifier (replace with nn.Identity()) after creation

Then any model(input) will output the embedding

What you proposed will work for some of the simpler models, but it will fail for many that have more than just a stand-alone global pool between the last feature layer and the classifier (a growing number of models). I am adding additional API forward_head() so that one can do the following as well in a way that will work for all models.

model = timm.create_model('resnet50')
unpooled_features = model.forward_features(img)
embeddings = model.forward_head(unpool_features, pre_logits=True)
predictions = model.forward_head(unpooled_features)

[strugoeli]

Thank you for your quick reply!

[rwightman]

No worries, thanks for the support!

[yonatanbitton]

Hey, thanks for the response and thanks for the awsome library as well :)
I am interested in this question as well.
I am training a classifier on top of the embeddings for some task.
I receive very different results between models that work with forward_features vs. models that work with the way you suggested here.

For example, comparing (ViT, DEiT) to (ResNet, EfficientNet):
in ViT & DEiT I execute:

vit_model = timm.create_model('vit_large_patch32_384', pretrained=True)
vit_img_featrues = vit_model.forward_features(img)

deit_model = timm.create_model('deit_base_patch16_384', pretrained=True)
deit_img_featrues = deit_model.forward_features(img)

In ResNet & EfficientNet:

resnet_model = timm.create_model('resnet101', num_classes=0, pretrained=True)
resnet_img_featrues = resnet_model.forward_features(img)

efficientnet_model = timm.create_model('efficientnet_b4', num_classes=0, pretrained=True)
efficientnet_model_features = efficientnet_model.forward_features(img)

I know that the models are different and that can explain the the results difference, however, I am afraid I am just using it wrong:
with ViT & DEiT I receive ~80% on my task, and with EfficientNet & ResNet I receive ~25% which is random baseline in my task.

Am I using it correctly? Am I missing something?

Thanks

[rwightman]

@yonatanbitton if you look at the shapes of those output they will be quite different between the vit and cnn models. To make them equivalent you'd want to call resnet_model(img) and use the normal forward (with the classifier removed as you have it)

forward_features on vit/deit/etc is currently returning a pooled (or token selected) output [B, num_features] , and on all of the convnets it's unpooled (roughly) [B, num_features, H//32, W//32]. To get [B, num_features] for convnets you should remove the classifer, keep pooling and call the normal forward...

I'm currently fixing that so it's more consistent between the model types, forward_features on transformer models will soon return the sequence so [B, L, num_features] and you'll need to call normal forward (w/ classifier removed) or forward_head w/ pre_logits set for the embeddings (after token selection or GAP)

[rwightman]

If you do this

vit_model = timm.create_model('vit_large_patch32_384', num_classes=0, pretrained=True)
vit_img_featrues = vit_model(img)

deit_model = timm.create_model('deit_base_patch16_384', num_clases=0, pretrained=True)
deit_img_featrues = deit_model(img)

resnet_model = timm.create_model('resnet101', num_classes=0, pretrained=True)
resnet_img_featrues = resnet_model(img)

efficientnet_model = timm.create_model('efficientnet_b4', num_classes=0, pretrained=True)
efficientnet_model_features = efficientnet_model(img)

they should all be consistent right now, the only thing that is currently inconsistent is forward_features between convnets and transformer/mlp models being unpooled for convnets and pooled (or token selected) for transformers and mlp, that will change so all transformer/mlp will return the 1d/2d token sequence (but you'd still have to know how to handle them yourself as they aren't 'equivalent' representations to the 2d convnet feature maps....

[yonatanbitton]

Thanks for the reply.
The question is how to achieve the behavior of (ViT, DEiT) with (ResNet, EfficientNet).

Just to clarify, this is how I use ViT (and it achieves 80% on my task)

vit_model = timm.create_model('vit_large_patch32_384', pretrained=True)
vit_img_features = vit_model.forward_features(img) # x.shape = torch.Size([1, 1024])

From your message:

you'd want to call resnet_model(img) and use the normal forward (with the classifier removed as you have it

I understand the following option 1 is the answer:

Option 1:

resnet_model = timm.create_model('resnet101', num_classes=0, pretrained=True)
resnet_img_features = resnet_model(img) # x.shape = torch.Size([1, 2048])  

However, there are other options as well:

Option 2:

resnet_model = timm.create_model('resnet101', num_classes=0, pretrained=True)
resnet_img_features = resnet_model.global_pool(resnet_model.forward_features(img)) # x.shape = torch.Size([1, 2048])  

Option 3:

resnet_model = timm.create_model('resnet101', pretrained=True)
resnet_img_features = resnet_model(img) # x.shape = torch.Size([1, 1000])  

Option 4:

resnet_model = timm.create_model('resnet101', pretrained=True)
resnet_img_features = resnet_model.global_pool(resnet_model.forward_features(img)) # x.shape = torch.Size([1, 2048])  

or is there other relevant options that I didn't take into account.
I actually tried all of these options and it doesn't seem to receive the same behavior as ViT, so I guess I am missing something.
Thanks again 🙏

[rwightman]

@yonatanbitton

option 2 works for resnet, but does not necessarily work for other networks as in the earlier discussion with OP above
option 3 is using the imagenet classifier output, which I don't think is what you want here
option 4 is the same as 2 but doesn't remove the classifier first (extra params hanging around, not a huge deal but DDP might complain)

Summary: option 1 (num_classes=0) is the simplest mechanism for getting pre-classifier embeddings that currently works across all model types. forward_features is supposed to provide unpooled features for all models, but currently an inconsistency between convnets and vision transformers (that I'll be fixing in next major merge).

[rwightman]

and FYI your original

resnet_model = timm.create_model('resnet101', num_classes=0, pretrained=True)
resnet_img_featrues = resnet_model.forward_features(img)

would have output [1, 2048, 7, 7] which would be quite different from vit/deit