Langchain Embedding with openapi and Pinecone vector dimensions not matching

[rmccorkl]

The embeddings appear successful

from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings( deployement="text-embedding-ada-002")

but the call to pinechain fails

import os
import pinecone
from langchain.vectorstores import Pinecone
pinecone.init(api_key=os.environ["PINECONE_API_KEY"], environment=os.environ["PINECONE_ENV"])
index_name1 = "bcexample"
search = Pinecone.from_documents(texts, embeddings, index_name=index_name1)

throwing the error

HTTP response body: {"code":3,"message":"Vector dimension 1536 does not match the dimension of the index 2048","details":[]}

[dosubot[bot]]

Answer generated by a 🤖

Hello,

The error you're encountering is due to a mismatch between the dimension of the embeddings generated by the OpenAIEmbeddings (1536) and the dimension of the Pinecone index (2048).

In the LangChain framework, when creating a new Pinecone index, the default dimension is set to 1536 to match the OpenAI embedding model text-embedding-ada-002 which uses 1536 dimensions. If you are using an existing Pinecone index with a different dimension, you will need to ensure that the dimension matches the dimension of the embeddings.

Here is the relevant code:

def add_texts(
    self,
    texts: Iterable[str],
    metadatas: Optional[List[dict]] = None,
    ids: Optional[List[str]] = None,
    namespace: Optional[str] = None,
    batch_size: int = 32,
    **kwargs: Any,
) -> List[str]:
    """Run more texts through the embeddings and add to the vectorstore.

    Args:
        texts: Iterable of strings to add to the vectorstore.
        metadatas: Optional list of metadatas associated with the texts.
        ids: Optional list of ids to associate with the texts.
        namespace: Optional pinecone namespace to add the texts to.

    Returns:
        List of ids from adding the texts into the vectorstore.
    """
    if namespace is None:
        namespace = self._namespace
    # Embed and create the documents
    docs = []
    ids = ids or [str(uuid.uuid4()) for _ in texts]
    embeddings = self._embed_documents(texts)
    for i, (text, embedding) in enumerate(zip(texts, embeddings)):
        metadata = metadatas[i] if metadatas else {}
        metadata[self._text_key] = text
        docs.append((ids[i], embedding, metadata))
    # upsert to Pinecone
    self._index.upsert(
        vectors=docs, namespace=namespace, batch_size=batch_size, **kwargs
    )
    return ids

Source: libs/langchain/langchain/vectorstores/pinecone.py

To resolve this issue, you need to ensure that the dimension of the embeddings matches the dimension of the Pinecone index. This could be achieved by either changing the dimension of the Pinecone index to match the dimension of the embeddings or by changing the dimension of the embeddings to match the dimension of the Pinecone index.

There was a similar issue in the LangChain repository: Issue: Chroma DB. The solution was to delete the collection and recreate it if an InvalidDimensionException was encountered. Although the issue was with Chroma DB, the solution might be applicable to your case as well.

I hope this helps! Let me know if you have any other questions.

Sources

• Issue: Chroma DB
• libs/langchain/langchain/vectorstores/pinecone.py
• docs/extras/integrations/vectorstores/pinecone.ipynb
• libs/langchain/tests/integration_tests/vectorstores/test_pinecone.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

If you you want to continue the conversation, start your reply with start your reply @dosu-bot. I'll only respond to replies from the issue author.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.