Responce includes Q&A

[TheJokerBG]

HI,
I'm useing a 32b Model witth Langchain and some HR docs.
Today i got a strange answer :
Question (by me) : What is the max credit for education we can get ?

Answer (by Model): Max credit is X$ per year. 

Question (by model as part of the above answer) : What is the process of requesting more credits ? 
Answer (by model as part of the above answer) : You need to contact HR and/or your team lead. 

It seems that it included a Q&A in the answer, obivously this is not part of the prompt template so is this a memory thing or is it related to another reason ?

[dosubot[bot]]

🤖

Hi,

The behavior you're observing is not a memory issue but rather a feature of the LangChain model. The model is designed to generate a Q&A within the response based on the prompt template defined in the PROMPT variable in the stuff_prompt.py file. This template guides the language model to generate a final answer and sources based on the provided document and question.

Here is the relevant code:

template = """Given the following extracted parts of a long document and a question, create a final answer with references ("SOURCES"). 
If you don't know the answer, just say that you don't know. Don't try to make up an answer.
ALWAYS return a "SOURCES" part in your answer.

QUESTION: {question}
=========
{summaries}
=========
FINAL ANSWER:"""
PROMPT = PromptTemplate(template=template, input_variables=["summaries", "question"])

In this template, {summaries} and {question} are placeholders that will be replaced with the actual document and question when generating a prompt. The "QUESTION:", "FINAL ANSWER:", and "SOURCES:" parts are cues for the model to start generating a question, an answer, and sources.

The LangChain model generates Q&A within the response by using a class called QAGenerateChain, which is a type of LLMChain. This class uses a prompt, which is a string that instructs the model to generate a final answer based on the context and the question. The QAGenerateChain class also uses an output parser, specifically a RegexParser, to parse the output of the language model. The RegexParser is configured to extract the question and answer from the output using a specific regex pattern. If the model doesn't know the answer, it is instructed to say that it doesn't know and not to make up an answer. The QAGenerateChain can be loaded from a BaseLanguageModel, which is the underlying language model used for generating the responses.

I hope this clarifies your question. If you have any more questions, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/chains/retrieval_qa/prompt.py
• libs/langchain/langchain/evaluation/qa/generate_prompt.py
• libs/langchain/langchain/chains/qa_with_sources/refine_prompts.py
• libs/langchain/langchain/chains/question_answering/refine_prompts.py
• libs/langchain/langchain/chains/qa_with_sources/stuff_prompt.py
• libs/langchain/langchain/evaluation/qa/generate_chain.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.