Custom Prompt from PromptTemplate and pass it to the RetrievalQA.

[mrutunjay-kinagi]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

def get_retrieval_chain () :
prompt_template = """     Role: You are an assistant for question-answering tasks.      Use the following pieces of retrieved context to answer the question.      If you don't know the answer , just say that you don't know or NA.      Keep the answer to the point follow the guidelines below.      Guidelines:     - If the query asks for a numerical value (e.g., amounts, percentages, dates), return only the relevant value.     - For yes/no questions, answer with either 'Yes' or 'No' based on the context.     - If no relevant information is available, return 'NA'.     - Do not provide any additional commentary or filler text. Focus on precision and brevity.     _ Use the following pieces of retrieved context to answer. 
Question:{query}      Context:{context}     """      


prompt = PromptTemplate(template=prompt_template, input_variables=["query","context"])

llm = BedrockChat(
        model_id="anthropic.claude-3-haiku-20240307-v1:0", region_name="us-east-1"
    )
    handler = StdOutCallbackHandler()
    vectorstore = get_opensearch_vectorstore(data, source)
    
    # create the retriever
    search_kwargs = {
        "filter": (
            {"term": {"metadata.source.keyword": source}} if source else {}
        ),  
        "top_k": 7, # Retrieves the top 7 most relevant documents  
    }

    print(f"search_kwargs: {search_kwargs}")
    retriever = vectorstore.as_retriever(search_kwargs=search_kwargs)
    # create the retriever chain
    qa_chain = RetrievalQA.from_chain_type(
        llm=llm,
        retriever=retriever,
        callbacks=[handler],
        return_source_documents=True,
        verbose=True,
        chain_type_kwargs={"prompt": prompt},
    )

    return qa_chain

qa_chain = get_retrieval_chain(data, source)
response = qa_chain.invoke({"query": prompt})

Description

I'm currently using langchain version 0.1.16 in my project. I'm not able to figure out how to pass the actual query asked by end user and the retrieved chunks as context to the PromptTemplate ?

System Info

langchain==0.1.16

[dosubot[bot]]

Hello, @mrutunjay-kinagi! I'm here to assist you with any bugs, questions, or contributions you may have.

To pass the actual query asked by the end user and the retrieved chunks as context to the PromptTemplate in LangChain version 0.1.16, you can use the PromptTemplate class. Here is an example of how you can achieve this:

from langchain.chains import RetrievalQA
from langchain_core.prompts import PromptTemplate
from langchain_community.llms import BedrockChat
from langchain_core.callbacks import StdOutCallbackHandler

def get_retrieval_chain(data, source):
    prompt_template = """
    Role: You are an assistant for question-answering tasks.
    Use the following pieces of retrieved context to answer the question.
    If you don't know the answer, just say that you don't know or NA.
    Keep the answer to the point follow the guidelines below.
    Guidelines:
    - If the query asks for a numerical value (e.g., amounts, percentages, dates), return only the relevant value.
    - For yes/no questions, answer with either 'Yes' or 'No' based on the context.
    - If no relevant information is available, return 'NA'.
    - Do not provide any additional commentary or filler text. Focus on precision and brevity.
    Use the following pieces of retrieved context to answer.
    Question: {query}
    Context: {context}
    """

    prompt = PromptTemplate(template=prompt_template, input_variables=["query", "context"])

    llm = BedrockChat(
        model_id="anthropic.claude-3-haiku-20240307-v1:0", region_name="us-east-1"
    )
    handler = StdOutCallbackHandler()
    vectorstore = get_opensearch_vectorstore(data, source)
    
    # create the retriever
    search_kwargs = {
        "filter": (
            {"term": {"metadata.source.keyword": source}} if source else {}
        ),  
        "top_k": 7, # Retrieves the top 7 most relevant documents  
    }

    retriever = vectorstore.as_retriever(search_kwargs=search_kwargs)
    
    # create the retriever chain
    qa_chain = RetrievalQA.from_chain_type(
        llm=llm,
        retriever=retriever,
        callbacks=[handler],
        return_source_documents=True,
        verbose=True,
        chain_type_kwargs={"prompt": prompt},
    )

    return qa_chain

qa_chain = get_retrieval_chain(data, source)
response = qa_chain.invoke({"query": "Your actual query here"})
print(response)

In this example, the PromptTemplate is created with a template that combines the query and the context into a standalone question. The RetrievalQA chain is then created using this prompt template, the language model, and the retriever. Finally, the chain is invoked with the user's query and the retrieved context ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

The values for the variables query and context are not directly passed in the line prompt = PromptTemplate(template=prompt_template, input_variables=["query", "context"]). Instead, they are placeholders within the template string that will be replaced with actual values when the PromptTemplate is used to generate a prompt. The actual values for query and context will be provided at runtime when the template is rendered with specific inputs.

For example, in the context_template:

context_template = """You are a teacher grading a quiz.
You are given a question, the context the question is about, and the student's answer. You are asked to score the student's answer as either CORRECT or INCORRECT, based on the context.

Example Format:
QUESTION: question here
CONTEXT: context the question is about here
STUDENT ANSWER: student's answer here
GRADE: CORRECT or INCORRECT here

Grade the student answers based ONLY on their factual accuracy. Ignore differences in punctuation and phrasing between the student answer and true answer. It is OK if the student answer contains more information than the true answer, as long as it does not contain any conflicting statements. Begin! 

QUESTION: {query}
CONTEXT: {context}
STUDENT ANSWER: {result}
GRADE:"""

The PromptTemplate is created as follows:

CONTEXT_PROMPT = PromptTemplate(
    input_variables=["query", "context", "result"], template=context_template
)

When this template is used, the placeholders {query}, {context}, and {result} will be replaced with actual values provided at that time ^[1].

To continue talking to Dosu, mention @dosu.

[mrutunjay-kinagi]

@dosu : Okay. In my case , context would be the retrieved chunks from vectorstore. And query would be the actual user query used to retrieve chunks. Will passing of these two values to the prompt llm post be taken care by langchain internally ?

[mikhailocampo]

@mrutunjay-kinagi To my knowledge this is the provided abstraction from a RetrievalQA chain. I recommend also checking out the runnable parallel concept. Think of it like taking the input and then copying it to pass on.

What I like to do so that I can understand conceptually under the hood is to use a Runnable Parallel with Runnable Passthrough like so,

retrieval = RunnableParallel(
  { "context": vector_store.as_retriever(), "query": RunnablePassthrough() }
).with_config({"run_name": "Retrieval Step"})

chain = (
  retrieval
  | PromptTemplate.from_template(template)
  | llm
  | StrOutputParser()
)

Then I can simply call the chain's chain.invoke("What is Langchain?"). This will create a copy of the query and pass to the vector store retriever to retrieve documents (chains just pipe the input and outputs together) while maintaining the original "query" field as "What is langchain?" Then, both of these are passed like a dict onto the prompt template which expects both "context" and "query" as input variables. From here, you can pass the prompt onto LLM and then parse the output to see the text in a pretty format.

Hope it helps!

[dosubot[bot]]

Yes, LangChain will internally handle passing the retrieved chunks from the vectorstore as context and the actual user query to the prompt LLM.

For example, using the RetrievalQA chain, LangChain manages the process of retrieving relevant chunks and passing them along with the user query to the LLM:

retriever = vectorstore.as_retriever()
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    chain_type_kwargs={"prompt": prompt},
)

response = qa_chain.invoke({"query": "Your actual query here"})

In this setup, the RetrievalQA chain takes care of retrieving the relevant chunks and combining them with the user query to generate the final prompt for the LLM ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[mrutunjay-kinagi]

@mikhailocampo: Thank you , Will try this out. The Example Code mentioned above is giving horrible results for the same dataset.