fine tuning CLIP for embedding generation

[imneonizer]

@ellie-sleightholm

Based on this notebook:
https://github.com/marqo-ai/fine-tuning-embedding-models-course/blob/main/10_fine_tuning_CLIP_models.ipynb

I went through the code, It's a nice write-up, kudos to the team!
However, features are calculated without gradients meaning only the last classification Linear node is actually being trained that too with fixed number of classes that makes it more like a How to train CLIP for image classification instead of finetuning the CLIP model and solving the original purpose of the CLIP model that is to compute custom embeddings aligned with the data for ranking purposes.

I would really appreciate If you can put a notebook to do that aswell.

[ellie-sleightholm]

Hey @imneonizer , thank you for your message. Absolutely! As we state in the article and Google Colab notebook, this is specific for image classification. We will work on a notebook for generally fine-tuning CLIP and update you when this is available 😊