Best practices for adding file dynamic in LlamaIndex with Chromadb

[FWiner]

Now we have a problem with this article.
That's my step:

1. Loading pdf file use SimpleDirectoryReader.
2. Create a Chroma collection and use ChromaVectorStore and BEG embeddings model to create index.

Now i want to add a new file in the rag system, and dynamic add the Documents or Nodes in persistent Chromadb, and update index directly.

[dosubot[bot]]

To dynamically add new files to a RAG system using LlamaIndex with ChromaDB and update the index directly, follow these steps:

1. Load the new file using SimpleDirectoryReader:

   from llama_index.core import SimpleDirectoryReader
   
   reader = SimpleDirectoryReader(input_files=["./path_to_your_pdf_file/your_file.pdf"])
   new_docs = reader.load_data()

2. Initialize the ChromaVectorStore with a persistent client:

   import chromadb
   from llama_index.vector_stores.chroma import ChromaVectorStore
   
   persist_dir = "path/to/chroma/directory"
   chroma_client = chromadb.PersistentClient(path=persist_dir)
   chroma_collection = chroma_client.get_or_create_collection("example_collection")
   vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

3. Convert the new documents to nodes:

   from llama_index.core.schema import TextNode
   
   new_nodes = [
       TextNode(
           text=doc.text,
           id_=f"new-id-{i}",
           relationships={},
           metadata=doc.metadata,
           embedding=[0.1, 0.2, 0.3],  # Replace with actual embeddings
       )
       for i, doc in enumerate(new_docs)
   ]

4. Add the new nodes to the ChromaDB and update the index:

   # Synchronous addition
   vector_store.add(new_nodes)
   
   # Asynchronous addition
   import asyncio
   asyncio.run(vector_store.async_add(new_nodes))

5. Query the updated index:

   from llama_index.core.vector_stores.types import VectorStoreQuery
   
   query_embedding = [0.1, 0.2, 0.3]  # Replace with actual query embedding
   res = vector_store.query(VectorStoreQuery(query_embedding=query_embedding, similarity_top_k=1))
   print(res.nodes[0].get_content())

6. Manage local index state (optional):
   If you need to manage local index state and ensure the index is updated correctly, you can use helper functions to read and update the local index state.

   • Read and update local index state:

     import pickle
     from pathlib import Path
     
     INDEXING_LOCAL_STATE_PATH = "path/to/local/state.pkl"
     
     def read_all_local_index_state():
         if not Path(INDEXING_LOCAL_STATE_PATH).is_file():
             return {}  # not found
     
         with open(INDEXING_LOCAL_STATE_PATH, "rb") as file:
             return pickle.load(file)
     
     def update_local_index(docset_id, full_doc_summaries_by_id, chunks_by_id, direct_tool_function_name, direct_tool_description, report_details):
         state = read_all_local_index_state()
     
         doc_index_state = {
             "full_doc_summaries_by_id": full_doc_summaries_by_id,
             "chunks_by_id": chunks_by_id,
             "retrieval_tool_function_name": direct_tool_function_name,
             "retrieval_tool_description": direct_tool_description,
             "reports": report_details,
         }
         state[docset_id] = doc_index_state
     
         with open(INDEXING_LOCAL_STATE_PATH, "wb") as file:
             pickle.dump(state, file)

   • Populate vector index:

     def populate_vector_index(docset_id, chunks, overwrite=False):
         persistent_client = chromadb.PersistentClient(path="path/to/chroma/directory")
     
         if overwrite:
             collections = persistent_client.list_collections()
             if any(c.name == docset_id for c in collections):
                 persistent_client.delete_collection(docset_id)
     
         collection = persistent_client.get_or_create_collection(docset_id)
         vector_store = ChromaVectorStore(chroma_collection=collection)
         storage_context = StorageContext.from_defaults(vector_store=vector_store)
     
         index = VectorStoreIndex.from_documents(chunks, storage_context=storage_context, embed_model="your_embedding_model")
         index.storage_context.persist(persist_dir="path/to/chroma/directory")

By following these steps, you can dynamically add new files to the RAG system, update the index directly using LlamaIndex with ChromaDB, and manage local index state effectively. This approach ensures that your index remains up-to-date with the latest documents and can be queried efficiently.

Details

Sources

• [Question]: How to use Auto merging retriever alongside chromadb
• [Question]: How to update existing index
• [Question]: Using Pinecone and S3 to update my index.
• Can not load it into Chroma, and then query it ！index.as_query_engine(chroma_collection=chroma_collection)
• Updating an index based on changes
• what happened to index.doc_previews (how should I list indexed documents?)
• llama-index-integrations/vector_stores/llama-index-vector-stores-chroma/tests/test_chromadb.py
• llama-index-packs/llama-index-packs-docugami-kg-rag/llama_index/packs/docugami_kg_rag/helpers/indexing.py
• llama-index-cli/llama_index/cli/rag/base.py
• llama-index-integrations/vector_stores/llama-index-vector-stores-chroma/llama_index/vector_stores/chroma/base.py
• llama-index-legacy/llama_index/legacy/node_parser/relational/hierarchical.py
• llama-index-core/llama_index/core/node_parser/relational/hierarchical.py
• docs/docs/examples/data_connectors/simple_directory_reader.ipynb
• llama-index-legacy/llama_index/legacy/readers/file/base.py
• llama-index-core/llama_index/core/readers/file/base.py
• docs/docs/module_guides/loading/simpledirectoryreader.md
• llama-index-legacy/llama_index/legacy/vector_stores/chroma.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.