chromadb.errors.NoIndexException: Index not found, please create an instance before querying

[allen-munsch]

What does this mean?

How can I load the following index?

tree  langchain/
langchain/
├── chroma-collections.parquet
├── chroma-embeddings.parquet
└── index
    ├── id_to_uuid_cfe8c4e5-8134-4f3d-a120-0510e189004f.pkl
    ├── index_cfe8c4e5-8134-4f3d-a120-0510e189004f.bin
    ├── index_metadata_cfe8c4e5-8134-4f3d-a120-0510e189004f.pkl
    └── uuid_to_id_cfe8c4e5-8134-4f3d-a120-0510e189004f.pkl

1 directory, 6 files

From the following code snippet:

if __name__ == "__main__":
    ABS_PATH = os.path.dirname(os.path.abspath(__file__))
    DB_DIR = os.path.join(ABS_PATH, 'langchain/index/index_cfe8c4e5-8134-4f3d-a120-0510e189004f.bin')
    client_settings = chromadb.config.Settings(
        chroma_db_impl="duckdb+parquet",
        persist_directory=DB_DIR,
        anonymized_telemetry=True
    )
    fp = './all_files.txt'
    embeddings = OpenAIEmbeddings()
    get_vectorstore = lambda: Chroma(
            collection_name="langchain",
            embedding_function=embeddings,
            client_settings=client_settings,
            persist_directory=DB_DIR,
        )

    if not os.path.exists(fp):
        root_dir = "."  # Replace with the desired root directory
        gitignore_path = os.path.join(root_dir, ".gitignore")
        ignored_patterns = get_ignored_patterns(gitignore_path)
        files = []
        concatenated_content = ""
        for file_path in find_files(root_dir, "*.py", ignored_patterns):
            files.append(file_path)
            with open(file_path, "r") as file:
                file_content = file.read()
                file_section = f"# <START> {file_path}\n{file_content}\n# <END> {file_path}\n"
                concatenated_content += file_section
        with open('all_files.txt', 'w') as f:
            f.write(concatenated_content)
        loader = TextLoader(fp)
        docs = []
        for loader in [loader]:
            docs.extend(loader.load())
        splitter = _get_default_text_splitter()
        sub_docs = splitter.split_documents(docs)
        vectorstore = get_vectorstore().from_documents(sub_docs, embeddings, persist_directory='./langchain', collection_name='langchain')
        index = VectorStoreIndexWrapper(vectorstore=get_vectorstore())
    else:
        # the defaults
        index = VectorStoreIndexWrapper(vectorstore=get_vectorstore())
    breakpoint()
    print(index)

python embed.py 
Using embedded DuckDB with persistence: data will be stored in: /home/jm/pycharm_projects/test/langchain/index/index_cfe8c4e5-8134-4f3d-a120-0510e189004f.bin
> /home/jm/pycharm_projects/test/embed.py(78)<module>()
-> print(index)
(Pdb) index.query('hello')
*** chromadb.errors.NoIndexException: Index not found, please create an instance before querying
(Pdb) 

Reference:

• https://stackoverflow.com/questions/76016847/chromadb-errors-noindexexception-index-not-found-please-create-an-instance-bef

[smpurkis]

I'm also having the same problem, reloading the chroma db from disk doesn't contain any index

[dholendar27]

from langchain.indexes import VectorstoreIndexCreator
index = VectorstoreIndexCreator().from_loaders([loader])

TRY THIS

[talk2jaydip]

Alternatively use GPTIndexing

def construct_index(directory_path):

set maximum input size

max_input_size = 4096

set number of output tokens

num_outputs = 256

set maximum chunk overlap

max_chunk_overlap = 20

set chunk size limit

chunk_size_limit = 600

prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)

define LLM

llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-davinci-003", max_tokens=num_outputs))

documents = SimpleDirectoryReader(directory_path).load_data()

service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
index = GPTSimpleVectorIndex.from_documents(documents, service_context=service_context)

index.save_to_disk('index.json')

return index

[Hamas-ur-Rehman]

@allen-munsch
whenever you create an index do this

docsearch = Chroma.from_texts([t.page_content for t in texts], embeddings, collection_name=collection_name , persist_directory=f'db/{collection_name}')
docsearch.persist()
docsearch = None

to load it try doing this

vectorstore = Chroma(collection_name=f'{collection_name}',persist_directory=f'./db/{collection_name}',embedding_function=embeddings)
qa = VectorDBQA.from_chain_type(llm=OpenAI(temperature=0.2), chain_type="map_reduce", vectorstore=vectorstore,return_source_documents=False)

[umair313]

this really worked for me. Thanks

btw why docsearch = None?

[Hamas-ur-Rehman]

its not necessary but if you are using multiple vect databases with the same variable then it might be a good practice