CLIP for image embedding

[alessandro-neri]

I want to create and index with Redis.
The index needs to store images, embedded with a custom model that I choose.

I tried with HuggingFaceEmbedding, or using SentenceTransformer externally but none of the 2 work.
MultimodalVectorStoreIndex does not seem to allow to choose the image model but uses only CLIP, correct?

How can I create a vector store for images (no text and images, only images) where I can choose the model (either a local modal or external APIs) ?

[jianliao]

Yes, you're correct that the default MultimodalVectorStoreIndex typically uses CLIP for embeddings, and you're restricted to the available CLIP models, as shown below:

AVAILABLE_CLIP_MODELS = (
    "RN50",
    "RN101",
    "RN50x4",
    "RN50x16",
    "RN50x64",
    "ViT-B/32",
    "ViT-B/16",
    "ViT-L/14",
    "ViT-L/14@336px",
)

If these models don’t meet your needs, you can extend the MultiModalEmbedding class to support any image embedding model you prefer.