Full Multimodal Support for Embedding Models and Vector Stores

[jakubbober]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Allow for usage of multimodal embeddings like Cohere Embed 4 and vector stores like PGVector

Motivation

When Cohere's Embed 4 multimodal embeddings came out, multimodal RAG/Vision RAG became the go-to solution for a lot of products. However, according to this page in LangChain documentation, the multimodal support exists only for chat models. The time has come to implement it all over the stack.

I love the simplicity of embedding documents and adding them to PGVector with LangChain (also using LlamaIndex's PDFReader in this particular example) as follows:

from langchain_postgres import PGVector
from llama_index.readers.file import PDFReader


embeddings = AzureOpenAIEmbeddings(deployment="text-embedding-3-large", api_version="2023-05-15")

vector_store = PGVector(
    embeddings=embeddings,
    collection_name="test",
    connection=os.getenv("POSTGRES_CONN_STR"),
    use_jsonb=True,
)

pages = PDFReader().load_data("file.pdf")
docs = [page.to_langchain_format() for page in pages]

vector_store.add_documents(docs)

And similarly fetching documents for RAG as follows:

docs_chunks = vector_store.similarity_search(query="hello!", k=5, search_type="similarity")

I'd really love to have the option to use multimodal embeddings like Cohere's Embed 4 in a similar simplistic fashion.

Proposal (If applicable)

No response