Question on text embeddings - why does this code not work properly?

[basementmedia2]

I use this code to create a vector database (chroma):

import os
cache_dir_temp = "/media/daten/huggingface_cache/"
os.environ['HF_HOME'] = cache_dir_temp
used_model = "flax-sentence-embeddings/all_datasets_v3_roberta-large"

from langchain.text_splitter import CharacterTextSplitter
from langchain_community.document_loaders import TextLoader
from langchain_chroma import Chroma
from langchain_huggingface import HuggingFaceEmbeddings

# Define the directory containing the text file and the persistent directory
current_dir = os.path.dirname(os.path.abspath(__file__))
file_path = os.path.join(current_dir, "books", "pottinger.txt")
persistent_directory = os.path.join(current_dir, "db", "chroma_db")

# Check if the Chroma vector store already exists
if not os.path.exists(persistent_directory):
    print("Persistent directory does not exist. Initializing vector store...")

    # Ensure the text file exists
    if not os.path.exists(file_path):
        raise FileNotFoundError(
            f"The file {file_path} does not exist. Please check the path."
        )

    # Read the text content from the file
    loader = TextLoader(file_path)
    documents = loader.load()

    # Split the document into chunks
    text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
    docs = text_splitter.split_documents(documents)

    # Display information about the split documents
    print("\n--- Document Chunks Information ---")
    print(f"Number of document chunks: {len(docs)}")
    print(f"Sample chunk:\n{docs[0].page_content}\n")

    # Create embeddings
    print("\n--- Creating embeddings ---")
    embeddings = HuggingFaceEmbeddings(
        model_name=used_model,
    )
    
    print("\n--- Finished creating embeddings ---")

    # Create the vector store and persist it automatically
    print("\n--- Creating vector store ---")
    db = Chroma.from_documents(
        docs, embeddings, persist_directory=persistent_directory)
    print("\n--- Finished creating vector store ---")

else:
    print("Vector store already exists. No need to initialize.")

And i use this code to find relevant text passages in the database:

import os
cache_dir_temp = "/media/daten/huggingface_cache/"
os.environ['HF_HOME'] = cache_dir_temp
used_model = "flax-sentence-embeddings/all_datasets_v3_roberta-large"


from langchain_chroma import Chroma
from langchain_huggingface import HuggingFaceEmbeddings


# Define the persistent directory
current_dir = os.path.dirname(os.path.abspath(__file__))
persistent_directory = os.path.join(current_dir, "db", "chroma_db")

# Define the embedding model
embeddings = HuggingFaceEmbeddings(
    model_name=used_model, 
)

# Load the existing vector store with the embedding function
db = Chroma(persist_directory=persistent_directory,
            embedding_function=embeddings)

# Define the user's question
query = "What is the name of the monkey?"

# Retrieve relevant documents based on the query
retriever = db.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={"k": 3, "score_threshold": 0.4},
)
relevant_docs = retriever.invoke(query)

# Display the relevant results with metadata
print("\n--- Relevant Documents ---")
for i, doc in enumerate(relevant_docs, 1):
    print(f"Document {i}:\n{doc.page_content}\n")
    if doc.metadata:
        print(f"Source: {doc.metadata.get('source', 'Unknown')}\n")

Creating the ChromaDB seems to work.
But althought the word "monkey" comes multiple times in the text file (pottinger.txt) it does not find relevant passages. Even if i set threshold to 0.2.

I have tested several embedding models from huggingface - always the same result.
What am i doing wrong?

Best wishes
Daniel

[dosubot[bot]]

Hey @basementmedia2! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together!

It seems like the issue might be related to how the embeddings are being generated or how the similarity search is configured. Here are a few things to check and try:

1. Embedding Model: Ensure that the embedding model you are using is suitable for your text data. Some models might not capture the nuances of certain types of text well. You might want to try different models that are specifically fine-tuned for sentence or document similarity tasks.

2. Embedding Function: Verify that the HuggingFaceEmbeddings is correctly generating embeddings. You can test this by manually embedding a sample text and checking the output.

3. Search Configuration: The search_type="similarity_score_threshold" and score_threshold=0.4 might be too strict. Try using a different search type, such as "similarity", and adjust the k value to see if it retrieves more results.

4. Document Splitting: Ensure that the document chunks are being created correctly. If the chunks are too small or not meaningful, it might affect the retrieval results. You can experiment with different chunk_size and chunk_overlap values.

5. Debugging: Add some debugging prints to check the embeddings and the similarity scores being calculated. This can help identify if the issue is with the embeddings or the retrieval process.

6. Model Cache: Make sure the model is correctly cached and loaded. Sometimes, issues with model loading can lead to unexpected results.

By checking these aspects, you should be able to identify where the problem lies and adjust accordingly.

To continue talking to Dosu, mention @dosu.