making llm responding geneal questions not related to context

[smartge3k]

I am using ConversationalRetrievalChain and passing it complete vector store from Pinecone as below:

qa = ConversationalRetrievalChain.from_llm(llm=OpenAI(temperature=0,openai_api_key=OPENAI_API_KEY),retriever=docsearch.as_retriever(search_kwargs={"k": 1}),chain_type="stuff",combine_docs_chain_kwargs={"prompt": QA},return_source_documents=True,verbose=True)

My LLM always gets 2 documents from the Vectorstore according to the query that is passed as context.

Now, my problem is I have hundreds of embeddings and each embedding discussed doing different tasks using different software (i.e how to register on Facebook, how to use Excel tables).
When I query the LLM "What can you help me with?" or "Which software do you know about?" or these general questions related to LLM knowledge base. It either responds with "I don't know" (as I have asked it to say I dont know if question is not related to the provided context)" or sometimes it will only say "I can help you with ABC or XYZ" ABC and XYZ are the two embeddings which are passed.

Is there a way that it can have knowledge of all the embeddings and can respond to such questions based on all the knowledge?
Below is my prompt

QA_PROMPT_DOCUMENT_CHAT = """
You are a  chatbot assistant. Your primary task is to answer questions based STRICTLY on the provided context. 

RULES:
- ONLY answer if the question relates directly to the provided context.
- Do NOT provide information that is not explicitly mentioned in the context. Avoid speculating or adding details from outside the context.
- If the question does NOT directly match with the context, respond with  I don't know.
- If no context is provided, always respond with I don't know.
- Always use more text to elaborate the answer. However, ensure the elaboration is strictly based on the context.
- If the question is a greeting (like "Hello", "Hi", etc.), respond with a corresponding greeting.

Remember: Stick to the context. If uncertain, respond with I don't know.

context: {context}

question: {question}
"""

[dosubot[bot]]

🤖

Based on your description, it seems like you want the language model to be able to answer general questions based on all the embeddings in the VectorStore, not just the two documents it retrieves based on the query.

There are two main ways to approach this:

1. Modify the prompt template: You can instruct the language model to use its own knowledge base when the context is not relevant. This was suggested in a similar issue in the LangChain repository (Issue: Not answering questions out of context using RetrievalQA Chain and ConversationalChatAgent). Here is the suggested code:

prompt_template = """If the context is not relevant, 
please answer the question by using your own knowledge about the topic

{context}

Question: {question}
"""

PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context", "question"]
)

chain_type_kwargs = {"prompt": PROMPT}
# Initialise Langchain - QA chain
qa = RetrievalQA.from_chain_type(llm=self.llm, 
                                 chain_type="stuff", 
                                 retriever=vectorstore.as_retriever(), 
                                 chain_type_kwargs=chain_type_kwargs)

2. Modify the _get_docs method: The _get_docs method in the ConversationalRetrievalChain class is responsible for retrieving the relevant documents based on the question. Currently, it retrieves only a limited number of documents (the top-k documents) that are most relevant to the question. To make it consider all embeddings, you need to modify this method to retrieve all documents from the VectorStore. Here is how you can modify the _get_docs method:

def _get_docs(
    self,
    question: str,
    inputs: Dict[str, Any],
    *,
    run_manager: CallbackManagerForChainRun,
) -> List[Document]:
    """Get docs."""
    # Retrieve all documents from the VectorStore
    docs = self.vectorstore.get_all_documents()
    return self._reduce_tokens_below_limit(docs)

Please note that this modification will significantly increase the computational cost and time, as the model will have to consider all documents in the VectorStore for each question. It may also decrease the relevance of the responses, as the model will consider a lot of potentially irrelevant information.

I hope this helps! Let me know if you have any other questions.

Sources

• Issue: Not answering questions out of context using RetrievalQA Chain and ConversationalChatAgent
• libs/langchain/langchain/chains/conversational_retrieval/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Alexong123]

So, do you have any solution for this situation? Because I also use ConversationalRetrievalChain and it also sometime give its own knowledge for the response

[smartge3k]

Unfortunately, no.

[Alexong123]

I see. In this case, may I ask if you have any alternative solutions if I really need to use an LLM model for conversation today?