Distilling features of SwinV2-L to SwinV2-T

[yopknopixx]

Hey guys, I am looking for some clarity on:

1. If Feature Distillation is suitable (beneficial) for my use case.
2. How do I go about the implementation.

Use case: I have trained a SwinV2-L model on a custom dataset and received decent results. But, I want to reduce the model size and inference time. Perhaps by using a smaller model like SwinV2-T.
In the repo, I can only see code for distilling ESVIT. So, I will have to add support for Swin as the teacher model.
Please let me know how to go about this and whether it is suited for my use case.

[aaparikh]

Same question