Conversational RAG tutorial on Langchain JS not working when it comes to out of context questions

[FaresKi]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

async discussWithLLM(history: ChatHistory, question: string) {
    const contextualizeQSystemPrompt = `Given a chat history and the latest user question
    which might reference context in the chat history, formulate a standalone question
    which can be understood without the chat history. Do NOT answer the question,
    just reformulate it if needed and otherwise return it as is. If the question is out of scope, please say that it is out of scope, and that you can't answer it.`;

    const qaSystemPrompt = `You are an assistant for question-answering tasks.
    Use the following pieces of retrieved context to answer the question.
    If you don't know the answer, just say that you don't know.
    Make sure to always back up your answers with quoted evidence from the context.
    If the question is out of scope, please say that it is out of scope, and that you can't answer it.
    {context}`;

    const contextualizeQPrompt = ChatPromptTemplate.fromMessages([
      ["system", contextualizeQSystemPrompt],
      new MessagesPlaceholder("chat_history"),
      ["human", "{question}"],
    ]);

    const contextualizeQChain = contextualizeQPrompt
      .pipe(this.openAI)
      .pipe(new StringOutputParser());

    const qaPrompt = ChatPromptTemplate.fromMessages([
      ["system", qaSystemPrompt],
      new MessagesPlaceholder("chat_history"),
      ["human", "{question}"],
    ]);

    const contextualizedQuestion = (input: {
      question: string;
      chat_history?: BaseMessageFields[];
    }): Runnable | string => {
      if ("chat_history" in input) {
        return contextualizeQChain;
      } else {
        return input.question;
      }
    };

    const retriever = this.vectorStore.asRetriever();

    const ragChain = RunnableSequence.from([
      RunnablePassthrough.assign({
        context: (input: {
          question: string;
          chat_history?: BaseMessageFields[];
        }) => {
          if ("chat_history" in input) {
            const chain = contextualizedQuestion(input) as Runnable;

            const response = chain
              .pipe(retriever)
              .pipe(formatDocumentsAsString);

            console.log("Contextualizing question...");
            return response;
          }
          return "";
        },
      }),
      qaPrompt,
      this.openAI,
    ]);

    const response = await ragChain.invoke({
      chat_history: history.map((chat) => {
        return chat.role === "user"
          ? new HumanMessage(chat.message)
          : new AIMessage(chat.message);
      }),
      question,
    });

    return response.content;
  }

Description

• I'm following the JS RAG tutorial for Conversational QA Chatbots here. So far it's been clear and I believe to have understood the basic components.
• I've checked the vector store and it embedded the source data correctly (a PDF file about atoms), but even out of context questions are answered (check screenshot):
  

What can I do to stop out of context questions from occuring?

System Info

[email protected]
platform (mac)
node 20.10.0
pnpm 9.4.0

[dosubot[bot]]

To stop out-of-context questions from being answered, you can use specific prompts designed to instruct the model to only answer questions based on the given context and to explicitly state if it does not know the answer. Here are some steps and code snippets to help you achieve this:

1. Use the DEFAULT_QA_PROMPT:
   This prompt ensures that the model only answers questions based on the provided context and explicitly states if it doesn't know the answer.

   import { PromptTemplate } from "@langchain/core/prompts";
   
   const DEFAULT_QA_PROMPT = new PromptTemplate({
     template: `Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\n{context}\n\nQuestion: {question}\nHelpful Answer:`,
     inputVariables: ["context", "question"],
   });

2. Update your qaPrompt to use DEFAULT_QA_PROMPT:
   Replace your current qaPrompt with the DEFAULT_QA_PROMPT to ensure the model follows the instructions.

   const qaPrompt = DEFAULT_QA_PROMPT;

3. Ensure the contextualizeQChain handles out-of-scope questions:
   Make sure the contextualizeQChain correctly identifies and handles out-of-scope questions.

   const contextualizeQSystemPrompt = `Given a chat history and the latest user question
   which might reference context in the chat history, formulate a standalone question
   which can be understood without the chat history. Do NOT answer the question,
   just reformulate it if needed and otherwise return it as is. If the question is out of scope, please say that it is out of scope, and that you can't answer it.`;
   
   const contextualizeQPrompt = ChatPromptTemplate.fromMessages([
     ["system", contextualizeQSystemPrompt],
     new MessagesPlaceholder("chat_history"),
     ["human", "{question}"],
   ]);
   
   const contextualizeQChain = contextualizeQPrompt
     .pipe(this.openAI)
     .pipe(new StringOutputParser());

4. Implement the formatDocumentsAsString function:
   Ensure you have the formatDocumentsAsString function to format the retrieved documents into a string.

   import { Document } from "@langchain/core/documents";
   
   const formatDocumentsAsString = (documents: Document[]): string =>
     documents.map((doc) => doc.pageContent).join("\n\n");

5. Update the ragChain to use the updated prompts and functions:
   Ensure your ragChain uses the updated qaPrompt and handles the context correctly.

   const ragChain = RunnableSequence.from([
     RunnablePassthrough.assign({
       context: (input: { question: string; chat_history?: BaseMessageFields[] }) => {
         if ("chat_history" in input) {
           const chain = contextualizedQuestion(input) as Runnable;
           const response = chain.pipe(retriever).pipe(formatDocumentsAsString);
           console.log("Contextualizing question...");
           return response;
         }
         return "";
       },
     }),
     qaPrompt,
     this.openAI,
   ]);

By following these steps and using the provided code snippets, you should be able to prevent out-of-context questions from being answered by ensuring the model only responds based on the given context and explicitly states when it doesn't know the answer ^[1].

To continue talking to Dosu, mention @dosu.

[FaresKi]

@dosu
Thanks for your answer. I do want to maintain history of the chat though. How can I do that?

[dosubot[bot]]

To maintain the chat history while ensuring out-of-context questions are not answered, you can use a sub-chain to reformulate the latest user question in the context of the chat history. This sub-chain will ensure that the question is understood without the chat history. Here is how you can implement it:

1. Define a sub-chain to contextualize the question:

import { ChatPromptTemplate, MessagesPlaceholder } from "@langchain/core/prompts";

const contextualizeQSystemPrompt = `Given a chat history and the latest user question
which might reference context in the chat history, formulate a standalone question
which can be understood without the chat history. Do NOT answer the question,
just reformulate it if needed and otherwise return it as is.`;

const contextualizeQPrompt = ChatPromptTemplate.fromMessages([
  ["system", contextualizeQSystemPrompt],
  new MessagesPlaceholder("chat_history"),
  ["human", "{question}"]
]);
const contextualizeQChain = contextualizeQPrompt.pipe(this.openAI).pipe(new StringOutputParser());

2. Use the sub-chain to reformulate follow-up questions:

import { AIMessage, HumanMessage } from "@langchain/core/messages";

await contextualizeQChain.invoke({
  chat_history: [
    new HumanMessage("What does LLM stand for?"),
    new AIMessage("Large language model") 
  ],
  question: "What is meant by large",
});

3. Build the full QA chain with chat history:

import { ChatPromptTemplate, MessagesPlaceholder } from "@langchain/core/prompts";
import { RunnablePassthrough, RunnableSequence } from "@langchain/core/runnables";
import { formatDocumentsAsString } from "langchain/util/document";

const qaSystemPrompt = `You are an assistant for question-answering tasks.
Use the following pieces of retrieved context to answer the question.
If you don't know the answer, just say that you don't know.
Use three sentences maximum and keep the answer concise.

{context}`;

const qaPrompt = ChatPromptTemplate.fromMessages([
  ["system", qaSystemPrompt],
  new MessagesPlaceholder("chat_history"),
  ["human", "{question}"]
]);

const contextualizedQuestion = (input: Record<string, unknown>) => {
  if ("chat_history" in input) {
    return contextualizeQChain;
  }
  return input.question;
};

const ragChain = RunnableSequence.from([
  RunnablePassthrough.assign({
    context: async (input: Record<string, unknown>) => {
      if ("chat_history" in input) {
        const chain = contextualizedQuestion(input);
        return chain.pipe(retriever).pipe(formatDocumentsAsString);
      }
      return "";
    },
  }),
  qaPrompt,
  this.openAI
]);

let chat_history = [];

const question = "What is task decomposition?";
const aiMsg = await ragChain.invoke({ question, chat_history });

console.log(aiMsg);

chat_history = chat_history.concat(aiMsg);

const secondQuestion = "What are common ways of doing it?";
await ragChain.invoke({ question: secondQuestion, chat_history });

This approach ensures that the chatbot only answers questions that are properly contextualized, thereby avoiding out-of-context responses while maintaining the chat history ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[FaresKi]

@dosu yes but in this case with chat history, how do i make sure that out of context questions aren't answered correctly
in my situation, i've ingested a PDF file about atoms, yet when asking what's the capital of France, I get "Paris" as an answer, instead of "i don't know, this question is out of context"

[dosubot[bot]]

To ensure that out-of-context questions are not answered while maintaining the chat history, you can modify your implementation to explicitly check if the question is relevant to the context. If the question is out of context, the system should respond accordingly. Here’s how you can do it:

1. Define the contextualizeQChain to reformulate questions:

   import { ChatPromptTemplate, MessagesPlaceholder } from "@langchain/core/prompts";
   import { StringOutputParser } from "@langchain/core/output_parsers";
   import { ChatOpenAI } from "@langchain/openai";
   
   const llm = new ChatOpenAI({ model: "gpt-3.5-turbo", temperature: 0 });
   
   const contextualizeQSystemPrompt = `Given a chat history and the latest user question
   which might reference context in the chat history, formulate a standalone question
   which can be understood without the chat history. Do NOT answer the question,
   just reformulate it if needed and otherwise return it as is. If the question is out of scope, please say that it is out of scope, and that you can't answer it.`;
   
   const contextualizeQPrompt = ChatPromptTemplate.fromMessages([
     ["system", contextualizeQSystemPrompt],
     new MessagesPlaceholder("chat_history"),
     ["human", "{question}"]
   ]);
   const contextualizeQChain = contextualizeQPrompt.pipe(llm).pipe(new StringOutputParser());

2. Modify the contextualizedQuestion function to handle out-of-context questions:

   const contextualizedQuestion = async (input: Record<string, unknown>) => {
     if ("chat_history" in input) {
       const reformulatedQuestion = await contextualizeQChain.invoke(input);
       if (reformulatedQuestion === "The question is out of scope, and that you can't answer it.") {
         return "The question is out of context and cannot be answered.";
       }
       return reformulatedQuestion;
     }
     return input.question;
   };

3. Update the ragChain to use the modified contextualizedQuestion function:

   import { RunnablePassthrough, RunnableSequence } from "@langchain/core/runnables";
   import { formatDocumentsAsString } from "langchain/util/document";
   import { MemoryVectorStore } from "langchain/vectorstores/memory";
   import { OpenAIEmbeddings } from "@langchain/openai";
   
   const vectorStore = new MemoryVectorStore(new OpenAIEmbeddings());
   const retriever = vectorStore.asRetriever();
   
   const qaSystemPrompt = `You are an assistant for question-answering tasks.
   Use the following pieces of retrieved context to answer the question.
   If you don't know the answer, just say that you don't know.
   Use three sentences maximum and keep the answer concise.
   
   {context}`;
   
   const qaPrompt = ChatPromptTemplate.fromMessages([
     ["system", qaSystemPrompt],
     new MessagesPlaceholder("chat_history"),
     ["human", "{question}"]
   ]);
   
   const ragChain = RunnableSequence.from([
     RunnablePassthrough.assign({
       context: async (input: Record<string, unknown>) => {
         if ("chat_history" in input) {
           const chain = await contextualizedQuestion(input);
           if (chain === "The question is out of context and cannot be answered.") {
             return chain;
           }
           return chain.pipe(retriever).pipe(formatDocumentsAsString);
         }
         return "";
       },
     }),
     qaPrompt,
     llm
   ]);

4. Invoke the ragChain with chat history:

   let chat_history = [];
   
   const question = "What is task decomposition?";
   const aiMsg = await ragChain.invoke({ question, chat_history });
   console.log(aiMsg);
   chat_history = chat_history.concat(aiMsg);
   
   const secondQuestion = "What are common ways of doing it?";
   const secondAiMsg = await ragChain.invoke({ question: secondQuestion, chat_history });
   console.log(secondAiMsg);

This implementation ensures that out-of-context questions are identified and not answered, while maintaining the chat history ^[1][2][3].

To continue talking to Dosu, mention @dosu.