Incorporate other embedding models such as DINOv2?

[YuanyuanLi96]

I enjoy using this library very much. However, I notice that other embedding techniques like DINOv2 may also be used in building the search index, and perhaps leads to higher retrieval accuracy. Is there an easy way I can load the 'facebook/dinov2-base' model from huggingface and still use clip_inference?

[ytzeng1]

One quick and dirty approach is just to load the state dict of DINOv2 to the visual encoder of a CLIP model, see the discuss in this thread if you are using open_clip. You probably need to retrain your text encoder in LIT style to align the text and image in the latent space if you wish to keep the text search functionality.