Chromadb embedding to FAISS.from_embeddings for query to document

[RickyGunawan09]

so i have a question, can i use embedding that i already store in chromadb and load it with faiss.from_embeddings ?

i already try it but i encounter some difficulty, this is how i try it:

check_chroma=Chroma.get(include=["embeddings","metadatas","documents"])
embeddings=check_chroma["embeddings"]
texts=check_chroma["documents"]

text_emb_pairs=zip(texts,embeddings)
multidocs_faiss=FAISS.from_embeddings(text_embeddings=text_emb_pairs,embedding=HuggingFaceEmbeddings())
retriever_multi_faiss=multidocs_faiss.as_retriever(search_type="similarity_score_threshold",search_kwargs={"score_threshold": 0.2, 'k': 3})

it throw error:

...\langchain\vectorstores\faiss.py", line 562, in __from
index = faiss.IndexFlatL2(len(embeddings[0]))
IndexError: list index out of range

can you tell me how to do it ?
thank you

[dosubot[bot]]

🤖

Hello,

From the error message you provided, it seems like the embeddings list is empty. The error is thrown when the code tries to determine the length of the first element of the embeddings list, but it can't because the list is empty.

The FAISS.from_embeddings method expects an iterable of tuples as input, where each tuple contains a string and a list of floats. The string represents the text, and the list of floats represents the corresponding embedding.

Here is the relevant part of the code:

@classmethod
def from_embeddings(
    cls,
    text_embeddings: Iterable[Tuple[str, List[float]]],
    embedding: Embeddings,
    metadatas: Optional[Iterable[dict]] = None,
    ids: Optional[List[str]] = None,
    **kwargs: Any,
) -> FAISS:
    """Construct FAISS wrapper from raw documents.

    This is a user friendly interface that:
        1. Embeds documents.
        2. Creates an in memory docstore
        3. Initializes the FAISS database

    This is intended to be a quick way to get started.

    Example:
        .. code-block:: python

            from langchain import FAISS
            from langchain.embeddings import OpenAIEmbeddings

            embeddings = OpenAIEmbeddings()
            text_embeddings = embeddings.embed_documents(texts)
            text_embedding_pairs = zip(texts, text_embeddings)
            faiss = FAISS.from_embeddings(text_embedding_pairs, embeddings)
    """
    texts = [t[0] for t in text_embeddings]
    embeddings = [t[1] for t in text_embeddings]
    return cls.__from(
        texts,
        embeddings,
        embedding,
        metadatas=metadatas,
        ids=ids,
        **kwargs,
    )

In your case, you are trying to load embeddings from ChromaDB and pass them to FAISS. The Chroma.get method returns a dictionary, and you are correctly extracting the embeddings and documents from this dictionary. However, it seems like the embeddings list is empty, which is causing the error.

Please double-check that the Chroma.get method is correctly retrieving the embeddings. You can do this by printing the length of the embeddings list after the Chroma.get call:

check_chroma=Chroma.get(include=["embeddings","metadatas","documents"])
embeddings=check_chroma["embeddings"]
print(len(embeddings))

If the length is 0, then the Chroma.get method is not retrieving the embeddings correctly. You might need to adjust the parameters of the Chroma.get call to correctly retrieve the embeddings.

I hope this helps! Let me know if you have any other questions.

Sources

• How to pass existing doc embeddings to FAISS ?
• libs/langchain/langchain/vectorstores/faiss.py
• libs/langchain/langchain/vectorstores/chroma.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[RickyGunawan09]

i read your code in faiss vectorsearch and i think i know why my embeddings is empty.

faiss.py#L641
faiss.py#L642

texts = [t[0] for t in text_embeddings] 
embeddings = [t[1] for t in text_embeddings]

this code i think is the problem ?.
the iterator is exhausted after the first iteration when you want to extract text, and there are no more elements left for the second iteration. As a result, embeddings is ends up being an empty list. and then i test it with simple python and yes it's become empty list.
here's my simple code.

a=[1,2,3,4]
b=["a","b","c","d"]
test=zip(a,b)
num=[t[0] for t in test]
var=[t[1] for t in test]
print(num)
print(var) #var become empty list

can you tell me how to make the embeddings not empty list ?
thank you