Getting embeddings of length 1 from OpenAIEmbeddings

[jaycresta]

Hi, I'm trying to use Langchain to create a vectorstore from scraped HTML pages, but I encountered an issue where I'm getting embeddings of length 1 when it should be 1536 per https://platform.openai.com/docs/guides/embeddings.

Here's how my code looks:

from langchain.document_loaders import BSHTMLLoader
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores.faiss import FAISS

...

all_raw_documents = []

for file in html_files:
    loader = BSHTMLLoader(file)
    raw_documents = loader.load()
    all_raw_documents.extend(raw_documents)

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
)
documents = text_splitter.split_documents(all_raw_documents)
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(documents, embeddings)

On the last line, I'm getting the below error:

Traceback (most recent call last):
  File "/ingest.py", line 128, in <module>
    ingest_docs(customers)
  File "/ingest.py", line 34, in ingest_docs
    vectorstore = FAISS.from_documents(documents, embeddings)
  File "/env/lib/python3.10/site-packages/langchain/vectorstores/base.py", line 272, in from_documents
    return cls.from_texts(texts, embedding, metadatas=metadatas, **kwargs)
  File "/env/lib/python3.10/site-packages/langchain/vectorstores/faiss.py", line 385, in from_texts
    return cls.__from(
  File "/env/lib/python3.10/site-packages/langchain/vectorstores/faiss.py", line 348, in __from
    index.add(np.array(embeddings, dtype=np.float32))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (13238,) + inhomogeneous part.

After some investigation, I found that the problem is due to getting the following erroneous embeddings:

{
  "embedding": [
    NaN
  ],
  "index": 520,
  "object": "embedding"
} 

As you can see, the embedding returned has length 1 instead of a real embedding with length 1536.

Does anyone know how to resolve this issue? Thanks!

[jaycresta]

For now we had to update /env/lib/python3.10/site-packages/langchain/embeddings/openai.py directly to retry embedding generation for the erroneous chunk inside _get_len_safe_embeddings().

[MariuszT]

Can you share your code? I have the same problem.

[luca-git]

Any official fix/workaround for that? I'm having the same issue.

[RasmusN]

Here is an ugly quick fix:
https://stackoverflow.com/questions/76469415/getting-embeddings-of-length-1-from-langchain-openaiembeddings/76475319#76475319

[fabianmax]

Works for me, thanks @RasmusN

[Holden-Lin]

for me, updating the langchain version solve this problem.
The new version has retry mechanism.

[kesun-ds]

May I ask what version you are upgrading to?

[kevinrossell]

0.0.240 worked for me

[luca-git]

The retry didn't work for me unless i set the chunk size in the low 00s. And I had to do the hardcode the fix. Talking about a few versions ago. The trouble is that, with chunk sizes in the 000s, there are almost always empty embeddings returned from opeanai. So it's pointless to retry 6 times, it always fails.

[nitinkr0411]

Oh ok, got it. Thanks :)

[Androbin]

Wouldn't it be more efficient to only recompute the embeddings that failed?

[luca-git]

@Androbin would be cool, how would you code that? My solution was quick and dirty hoping for some official fix, but not sure it's there.

[Androbin]

@luca-git Here is my solution: batched, nan-safe, rate-limit-safe
https://gist.github.com/Androbin/7635930c49f44072ffc36cc463c180ac

[luca-git]

🙏