Embedding model works but model does not see the information pulled up from the vector database

[EXXPRT]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.llms import Ollama
from langchain.prompts import PromptTemplate
from langchain.chains import RetrievalQA
from langchain.vectorstores import Chroma
from langchain.embeddings import OllamaEmbeddings

# Initialize Ollama embeddings and LLM
ollama_embeddings = OllamaEmbeddings(model="bge-m3:latest", base_url="http://localhost:11434")
ollama_llm = Ollama(model="gemma3:27b", base_url="http://localhost:11434")

# Load the vector store from disk
chroma_db = Chroma(persist_directory="./text2/chroma_db", embedding_function=ollama_embeddings)

# Define a prompt template
prompt_template = PromptTemplate(
    input_variables=["context", "question"],
    template="From the context below, extract the exact number for the question. If not found, say 'Not Available'. Context: {context}. Question: {question}."
)

# Create the RetrievalQA chain
qa_chain = RetrievalQA.from_chain_type(
    llm=ollama_llm,
    retriever=chroma_db.as_retriever(search_kwargs={"k": 300}),
    chain_type_kwargs={"prompt": prompt_template}
)

# Function to answer questions with debug output
def answer_question(question):
    # Retrieve context for debugging
    docs = chroma_db.similarity_search(question, k=300)
    retrieved_context = "\n".join([doc.page_content for doc in docs])
    print("Retrieved Context:\n", retrieved_context)
    
    result = qa_chain({"query": question})
    return result["result"]

# Example usage
question = "how many incidents were there in 2019?"
answer = answer_question(question)
print(f"Question: {question}")
print(f"Answer: {answer}")

Description

Hi guys, i am working on this RAG implementation running on Langchain, Ollama and ChromaDB.

preface: this is a smaller version of the workflow that I am running for my own testing, my desktop specifications are not ideal but ill have to make do.

The retrival from the vector database using BGE-m3 is working fine. The issue arises when it comes to the model answering the question based on the information from the vector database. The picture below shows the response from the model.

do let me know if there are any modifications i can make to the code to improve the model reasoning and answering.

my initial thoughts on this issue:

• model is not receiving information from the vector database
• model is not good enough (interms of generalization)
• model has a small context length that does not allow large amount of information to be taken into account

System Info

AMD 6800
i5-10400F
16 GB DDR4