Chroma.from_documents Segmentation Fault

[MaximeCarriere]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

def loader(link):
    loader = WebBaseLoader(link)
    data = loader.load()
    return data

def splitter(data):
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1500, chunk_overlap=100)
    all_splits = text_splitter.split_documents(data)
    return all_splits

def vector_storage(all_splits, name):
    vectorstore = Chroma.from_documents(documents=all_splits,
                                    embedding=GPT4AllEmbeddings(), 
                                    persist_directory="./chroma_data/"+name)
    return vectorstore

QA_CHAIN_PROMPT = hub.pull("rlm/rag-prompt-llama")
llm = load_llm()

for i in df["websites"]:
 

    try:
        data = loader(i)
        all_splits = splitter(data)
        vectorstore = vector_storage(all_splits, i)
        #question = "Can you give me a summary of this webpage"# the field of this company"
        docs = vectorstore.similarity_search(question)
        qa_chain = get_qa_chain(llm, QA_CHAIN_PROMPT, vectorstore)
        result = qa_chain({"query": question})
        print(result)
    except:
        print(i + "   not working")

Description

I am a new user of Chroma and I have some difficulties concerning Chroma.from_documents.

I have a simple code running.
It works quite well but after a couple of iteration I got a segmentaion fault.

I saw this in the documentation of Chroma:

Caution: Chroma makes a best-effort to automatically save data to disk, however multiple in-memory clients can stomp each other’s work. As a best practice, only have one client per path running at any given time.

However I am not quite sure what is wrong? Should I not use the Chroma.from_documents this way? Should I create first a collection of all documents and then ask in the for loop to check specifically to certain documents? Any other recommandations?

Thank you for your help !

System Info

langchain==0.1.0
langchain-community==0.0.12
langchain-core==0.1.10

[dosubot[bot]]

🤖

Hello again @MaximeCarriere! Good to see you back. It appears you've encountered a new challenge with LangChain. Let's see what we can do about it.

The Chroma.from_documents method is used to create a Chroma vectorstore from a list of documents. It takes a list of documents, an optional embedding function, optional list of document IDs, a collection name, an optional persist directory, optional client settings, an optional client, and optional collection metadata as arguments. If a persist directory is specified, the collection will be persisted there. Otherwise, the data will be ephemeral in-memory. It creates a new instance of the Chroma class with the provided arguments. If the list of document IDs is not provided, it generates a new list of unique IDs for each document. It then adds the documents to the Chroma collection.

Regarding the segmentation fault you're experiencing, it's hard to pinpoint the exact cause without more information. However, the caution you mentioned from the Chroma documentation suggests that having multiple clients accessing the same path at the same time can cause issues. In your case, it seems like you're creating a new Chroma instance for each website in your loop. If these instances are trying to access the same persist directory at the same time, that could potentially cause a segmentation fault.

One possible solution could be to create a single Chroma instance outside of your loop and add documents to it within the loop. This way, you're not creating multiple instances that could potentially conflict with each other. Here's an example of how you might do this:

vectorstore = Chroma(embedding=GPT4AllEmbeddings(), persist_directory="./chroma_data/")

for i in df["websites"]:
    try:
        data = loader(i)
        all_splits = splitter(data)
        vectorstore.add_documents(documents=all_splits, ids=[i]*len(all_splits))
        # rest of your code
    except:
        print(i + "   not working")

In this example, add_documents is a hypothetical method that adds documents to an existing Chroma instance. You'll need to replace it with the actual method used in LangChain for adding documents to a Chroma instance.

Please note that this is just a suggestion based on the information available. If this doesn't solve your issue, could you please provide more details about the segmentation fault? For example, the exact error message and the point in your code where it occurs would be helpful.

Sources

• libs/langchain/langchain/vectorstores/chroma.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Richard-Lu-badbird]

Hi there, it comes to me the same question segmentation defalut. After my check, when i run this line of python code, it will hint me 'Segmentation fault'.vectorstore = Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings()).So. how did you solve the question? I search the whole internet and find no answer....Boom！

[yangguoer]

请问您解决了吗，我也遇到了同样的问题

[PapyrusNotes]

For more details, terminal session executing python -X faulthandler -m streamlit run main.py --logger.level=debug says
"Window fatal exception: access violation" proceeded by "Thread {thread id}".

it occurs in following packages : tqdm , posthog, and chromadb