Get (document, similarity score) pairs from retriever #4595
-
Checked other resources
Commit to Help
Example Codepublic static createConversationalRetrievalChain(
args: ConversationalRetrievalChainParams,
onDocumentsRetrieved: (documents: Document[]) => void,
): RunnableSequence {
const { llm, retriever } = args;
const condenseQuestionTemplate = `Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.
Chat History:
{chat_history}
Follow Up Input: {question}
Standalone question:`;
const CONDENSE_QUESTION_PROMPT = PromptTemplate.fromTemplate(
condenseQuestionTemplate
);
const answerTemplate = `Answer the question with as detailed as possible based only on the following context:
{context}
Question: {question}
`;
const ANSWER_PROMPT = PromptTemplate.fromTemplate(answerTemplate);
const formatChatHistory = (chatHistory: [string, string][]) => {
const formattedDialogueTurns = chatHistory.map(
(dialogueTurn) => `Human: ${dialogueTurn[0]}\nAssistant: ${dialogueTurn[1]}`
);
return formattedDialogueTurns.join("\n");
};
const standaloneQuestionChain = RunnableSequence.from([
{
question: (input: ConversationalRetrievalQAChainInput) => input.question,
chat_history: (input: ConversationalRetrievalQAChainInput) =>
formatChatHistory(input.chat_history),
},
CONDENSE_QUESTION_PROMPT,
llm,
new StringOutputParser(),
]);
const formatDocumentsAsStringAndStore = async (documents: Document[]) => {
// Store or log documents for debugging
onDocumentsRetrieved(documents);
return formatDocumentsAsString(documents);
};
const answerChain = RunnableSequence.from([
{
context: retriever.pipe(formatDocumentsAsStringAndStore),
question: new RunnablePassthrough(),
},
ANSWER_PROMPT,
llm,
]);
const conversationalRetrievalQAChain = standaloneQuestionChain.pipe(answerChain);
return conversationalRetrievalQAChain as RunnableSequence;
}DescriptionNeed guidance on getting the (doc, similarity score) pairs from this LCEL setup. I have a callback System Infonode v19.8.1 |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 5 replies
-
|
🤖 Hey @logancyang, great to see you diving into another challenge with langchainjs! Let's tackle this one together. 🚀 Based on the code you've provided, it seems you're trying to retrieve both the documents and their similarity scores from the retriever. However, the current implementation of the To achieve this, you would need to modify or extend the retriever to also return the similarity scores. Here's a general approach:
class ScoreThresholdRetrieverWithScores extends ScoreThresholdRetriever {
async getRelevantDocumentsWithScores(query: string): Promise<Array<{document: Document, score: number}>> {
// Your implementation here
}
}
Please note that this is a general approach and the actual implementation would depend on the internals of the For more information on the structure of the I hope this helps! If you have any more questions or need further clarification, feel free to ask. Sources
|
Beta Was this translation helpful? Give feedback.
-
|
@dosu-bot Assuming that I implemented this |
Beta Was this translation helpful? Give feedback.
-
|
🤖 To integrate the First, ensure that your Then, modify the
Here's an example modification of the async _call(
values: ChainValues,
runManager?: CallbackManagerForChainRun
): Promise<ChainValues> {
if (!(this.inputKey in values)) {
throw new Error(`Question key ${this.inputKey} not found.`);
}
if (!(this.chatHistoryKey in values)) {
throw new Error(`Chat history key ${this.chatHistoryKey} not found.`);
}
const question: string = values[this.inputKey];
const chatHistory: string =
ConversationalRetrievalQAChain.getChatHistoryString(
values[this.chatHistoryKey]
);
let newQuestion = question;
if (chatHistory.length > 0) {
const result = await this.questionGeneratorChain.call(
{
question,
chat_history: chatHistory,
},
runManager?.getChild("question_generator")
);
const keys = Object.keys(result);
if (keys.length === 1) {
newQuestion = result[keys[0]];
} else {
throw new Error(
"Return from llm chain has multiple values, only single values supported."
);
}
}
// Use getRelevantDocumentsWithScores if available
let docs;
if (typeof this.retriever.getRelevantDocumentsWithScores === 'function') {
const { documents, scores } = await this.retriever.getRelevantDocumentsWithScores(
newQuestion,
runManager?.getChild("retriever")
);
docs = documents; // Assuming you want to proceed with documents only
// Handle scores if needed
} else {
docs = await this.retriever.getRelevantDocuments(
newQuestion,
runManager?.getChild("retriever")
);
}
const inputs = {
question: newQuestion,
input_documents: docs,
chat_history: chatHistory,
};
let result = await this.combineDocumentsChain.call(
inputs,
runManager?.getChild("combine_documents")
);
if (this.returnSourceDocuments) {
result = {
...result,
sourceDocuments: docs,
};
}
if (this.returnGeneratedQuestion) {
result = {
...result,
generatedQuestion: newQuestion,
};
}
return result;
}This example assumes that Please note that the You can refer to the source code of the Sources
|
Beta Was this translation helpful? Give feedback.
-
|
Hey @logancyang, this is something we've been looking into a bit. The tricky thing is that we don't want to store scores as metadata on the documents themselves since those can be resaved to a store later. If you're ok with that happening, you could create your own custom retriever: https://js.langchain.com/docs/modules/data_connection/retrievers/custom Or you could just wrap the vectorstore directly in something like a |
Beta Was this translation helpful? Give feedback.
-
|
@jacoblee93 thanks for the answer! Do you guys have plans to add the ensemble retriever https://python.langchain.com/docs/modules/data_connection/retrievers/ensemble to the JS version too? |
Beta Was this translation helpful? Give feedback.
-
|
Nothing immediate but we'd welcome a community contribution! |
Beta Was this translation helpful? Give feedback.
-
|
@dosu-bot actually, how do I add console.log to show the output of |
Beta Was this translation helpful? Give feedback.
-
|
Based on the code you've provided, it seems like you're trying to debug the process of generating a standalone question from a follow-up question within a conversation context. To do this, you can add Here's how you can do it:
const standaloneQuestionChain = RunnableSequence.from([
{
question: (input: ConversationalRetrievalQAChainInput) => {
console.log("Input Question: ", input.question);
return input.question;
},
chat_history: (input: ConversationalRetrievalQAChainInput) => {
const formattedChatHistory = formatChatHistory(input.chat_history);
console.log("Formatted Chat History: ", formattedChatHistory);
return formattedChatHistory;
},
},
CONDENSE_QUESTION_PROMPT,
llm,
new StringOutputParser(),
]);
const answerChain = RunnableSequence.from([
{
context: retriever.pipe(formatDocumentsAsStringAndStore),
question: new RunnablePassthrough((question) => {
console.log("Question for Answer Chain: ", question);
return question;
}),
},
ANSWER_PROMPT,
llm,
]);These Remember to remove or comment out the For more information on how the If you need further assistance, feel free to ask. Sources
|
Beta Was this translation helpful? Give feedback.
-
|
@dosu-bot how come I do not see the console log from the 2nd snippet you suggested, but can see them from the 1st snippet successfully? This is the snippet from which I do not see the message logged |
Beta Was this translation helpful? Give feedback.
-
|
Based on the provided context and the code snippet, it seems like you have correctly implemented the However, if you're not seeing the console log message "Question for Answer Chain: ", it could be due to a few reasons:
To debug this further, you could add more logging at different points in the sequence to see how far the input is getting. For example, you could add a const answerChain = RunnableSequence.from([
new RunnablePassthrough((input) => {
console.log("Initial input: ", input);
return input;
}),
{
context: retriever.pipe(formatDocumentsAsStringAndStore),
question: new RunnablePassthrough((question) => {
console.log("Question for Answer Chain: ", question);
return question;
}),
},
ANSWER_PROMPT,
llm,
]);This will help you determine whether the input is reaching the sequence and how it's being transformed as it moves through the sequence. If you're still having trouble, please provide more details about your runtime environment and the context in which you're invoking the Sources
|
Beta Was this translation helpful? Give feedback.
Hey @logancyang, this is something we've been looking into a bit. The tricky thing is that we don't want to store scores as metadata on the documents themselves since those can be resaved to a store later.
If you're ok with that happening, you could create your own custom retriever:
https://js.langchain.com/docs/modules/data_connection/retrievers/custom
Or you could just wrap the vectorstore directly in something like a
RunnableLambdaand return tuples.