How do I turn off rephrase question in ConversationalRetrievalQA?

[KennyJinhiro]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import { ChatOpenAI } from "langchain/chat_models/openai";
import { ConversationalRetrievalQAChain } from "langchain/chains";
import { HNSWLib } from "langchain/vectorstores/hnswlib";
import { OpenAIEmbeddings } from "langchain/embeddings/openai";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { BufferMemory } from "langchain/memory";

import * as fs from "fs";

export const run = async () => {
  const text = fs.readFileSync("state_of_the_union.txt", "utf8");
  const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000 });
  const docs = await textSplitter.createDocuments([text]);
  const vectorStore = await HNSWLib.fromDocuments(docs, new OpenAIEmbeddings());
  const fasterModel = new ChatOpenAI({
    modelName: "gpt-3.5-turbo",
  });
  const slowerModel = new ChatOpenAI({
    modelName: "gpt-4",
  });
  const chain = ConversationalRetrievalQAChain.fromLLM(
    slowerModel,
    vectorStore.asRetriever(),
    {
      returnSourceDocuments: true,
      memory: new BufferMemory({
        memoryKey: "chat_history",
        inputKey: "question", // The key for the input to the chain
        outputKey: "text", // The key for the final conversational output of the chain
        returnMessages: true, // If using with a chat model (e.g. gpt-3.5 or gpt-4)
      }),
      questionGeneratorChainOptions: {
        llm: fasterModel,
      },
    }
  );
  /* Ask it a question */
  const question = "What did the president say about Justice Breyer?";
  const res = await chain.call({ question });
  console.log(res);

  const followUpRes = await chain.call({ question: "Was that nice?" });
  console.log(followUpRes);
};

Description

I am trying to turn rephrase question off as it always gives wrong answers to greetings and gratitudes.

System Info

Node version
Mac

[dosubot[bot]]

Hey @jinhro! I'm here to help you with your questions and bugs. I can also guide you on how to contribute. If you need any assistance, feel free to ask!

To turn off the rephrase question feature in ConversationalRetrievalQAChain, you need to modify the questionGeneratorChainOptions to not include a template for rephrasing. Here is how you can do it:

import { ChatOpenAI } from "langchain/chat_models/openai";
import { ConversationalRetrievalQAChain } from "langchain/chains";
import { HNSWLib } from "langchain/vectorstores/hnswlib";
import { OpenAIEmbeddings } from "langchain/embeddings/openai";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { BufferMemory } from "langchain/memory";

import * as fs from "fs";

export const run = async () => {
  const text = fs.readFileSync("state_of_the_union.txt", "utf8");
  const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000 });
  const docs = await textSplitter.createDocuments([text]);
  const vectorStore = await HNSWLib.fromDocuments(docs, new OpenAIEmbeddings());
  const fasterModel = new ChatOpenAI({
    modelName: "gpt-3.5-turbo",
  });
  const slowerModel = new ChatOpenAI({
    modelName: "gpt-4",
  });
  const chain = ConversationalRetrievalQAChain.fromLLM(
    slowerModel,
    vectorStore.asRetriever(),
    {
      returnSourceDocuments: true,
      memory: new BufferMemory({
        memoryKey: "chat_history",
        inputKey: "question", // The key for the input to the chain
        outputKey: "text", // The key for the final conversational output of the chain
        returnMessages: true, // If using with a chat model (e.g. gpt-3.5 or gpt-4)
      }),
      questionGeneratorChainOptions: {
        llm: fasterModel,
        template: "" // This disables the rephrase question feature
      },
    }
  );
  /* Ask it a question */
  const question = "What did the president say about Justice Breyer?";
  const res = await chain.call({ question });
  console.log(res);

  const followUpRes = await chain.call({ question: "Was that nice?" });
  console.log(followUpRes);
};

By setting the template to an empty string in questionGeneratorChainOptions, you effectively disable the rephrase question feature ^[1].

To continue talking to Dosu, mention @dosu.

[KennyJinhiro]

@dosu How do I add history in RetrievalQA, rather than conversationalretrievalQA?

[dosubot[bot]]

To add history in RetrievalQA rather than ConversationalRetrievalQA, you can use the createHistoryAwareRetriever function. Here is an example of how you can do it:

// yarn add langchain @langchain/openai

import { ChatOpenAI } from "@langchain/openai";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { MemoryVectorStore } from "langchain/vectorstores/memory";
import { createHistoryAwareRetriever } from "langchain/chains/history_aware_retriever";
import { createRetrievalChain } from "langchain/chains/retrieval";
import { RunnablePassthrough, RunnableSequence } from "@langchain/core/runnables";
import { StringOutputParser } from "@langchain/core/output_parsers";

const QUESTION_GEN_TEMPLATE = `Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question.

Chat History:
{chat_history}
Follow Up Input: {input}
Standalone question:`;

const COMBINE_DOCS_PROMPT = `Based on the following context:

{context}

And chat history:
{chat_history}

Answer the following question:
{input}`;

const questionGenPrompt = ChatPromptTemplate.fromTemplate(QUESTION_GEN_TEMPLATE);
const vectorstore = await MemoryVectorStore.fromTexts(
  [
    "Mitochondria is the powerhouse of the cell",
    "Foo is red",
    "Bar is red",
    "Buildings are made out of brick",
    "Mitochondria are made of lipids",
  ],
  [{ id: 2 }, { id: 1 }, { id: 3 }, { id: 4 }, { id: 5 }],
  new OpenAIEmbeddings()
);
const retriever = vectorstore.asRetriever(2);
const llm = new ChatOpenAI({});
const historyAwareRetriever = await createHistoryAwareRetriever({
  llm,
  retriever,
  rephrasePrompt: questionGenPrompt,
});
const combineDocsPrompt = ChatPromptTemplate.fromTemplate(COMBINE_DOCS_PROMPT);
const combineDocsChain = RunnableSequence.from([
  RunnablePassthrough.assign({
    context: (input: { context: Document[] }) =>
      input.context.map((doc) => doc.pageContent).join("\n\n"),
  }),
  combineDocsPrompt,
  llm,
  new StringOutputParser(),
]);
const chain = await createRetrievalChain({
  retriever: historyAwareRetriever,
  combineDocsChain,
});

const result = await chain.invoke({
  input: "What is the powerhouse of the cell?",
  chat_history: "",
});

console.log(result);

const result2 = await chain.invoke({
  input: "What are they made of?",
  chat_history: [
    "Human: What is the powerhouse of the cell?",
    "Assistant: Mitochondria is the powerhouse of the cell",
  ].join("\n"),
});

console.log(result2);

In this example, the createHistoryAwareRetriever function is used to create a chain that can handle conversation history. The chain.invoke method takes an object with input and chat_history properties, allowing you to pass the conversation history to the retriever ^[1][2].

To continue talking to Dosu, mention @dosu.