How can I replicate the load_qa_with_sources_chain() from the python library in the javascript library?

[Simbaclaws]

Hi there,

For my web application I'm trying to use faiss with source documents that have url's to a web page. These url's should be mentioned as source inside the answer of what my gpt is going to give me back. My gpt should search all of the documents that it can find in a chain, and create an answer based on all of the provided documents from a single or sometimes multiple sources.

I haven't seen any load_qa_with_sources_chain equivalents of the python library that does just that.

So I found some code somewhere (can't remember where, sorry for the original author) that should replicate this functionality the way I want it to:

const vectorstore = await FaissStore.loadFromPython("./knowledge", new OpenAIEmbeddings({
            modelName: "text-embedding-ada-002"
}))

const vectorStoreRetriever = vectorstore.asRetriever();
// Create a system & human prompt for the chat model
const SYSTEM_TEMPLATE = `
My custom prompt that I'm giving to my gpt

==========
{question}
==========

Answer:
Source(s):`;

const templateMessages = [
    SystemMessagePromptTemplate.fromTemplate(SYSTEM_TEMPLATE),
    HumanMessagePromptTemplate.fromTemplate("{question}"),
];

const docs = await vectorstore.similaritySearch(question, 1);
const prompt = ChatPromptTemplate.fromMessages(templateMessages);

const chain = RunnableSequence.from([
    {
        // Extract the "question" field from the input object and pass it to the retriever as a string
        sourceDocuments: RunnableSequence.from([
            (input) => input.question,
            vectorStoreRetriever,
        ]),
        question: (input) => input.question,
    },
    {
        // Pass the source documents through unchanged so that we can return them directly in the final result
        sourceDocuments: (previousStepResult) => previousStepResult.sourceDocuments,
        question: (previousStepResult) => previousStepResult.question,
        context: (previousStepResult) =>
            formatDocumentsAsString(previousStepResult.sourceDocuments),
    },
    {
        result: prompt.pipe(model).pipe(new StringOutputParser()),
        sourceDocuments: (previousStepResult) => previousStepResult.sourceDocuments,
    },
]);

const response = await chain.invoke({
    question: question,
    input_documents: docs
})

However, currently my gpt is responding with just the last sourceDocument in the sourceDocuments array. As well as not return any of the sources in the answer it should generate....

Does anyone know what I am doing wrong in order to get my gpt to respond based on all of the sourceDocuments in the chain, as well as with the correct source url's?

Please help me solve this last issue I'm having.

[dosubot[bot]]

🤖

Hello @Simbaclaws,

Nice to meet you! I'm Dosu, a bot designed to assist with issues and questions regarding the langchainjs repository, as well as help guide contributors. While we're waiting for a human maintainer to respond, feel free to ask me any questions you may have. I'll do my best to help you.

Based on the context you've provided, it seems like you're trying to replicate the functionality of load_qa_with_sources_chain() from the Python library in JavaScript. The code you've shared is on the right track, but it seems like there might be an issue with how the source documents are being formatted and passed to the model.

In the current setup, the formatDocumentsAsString() function is used to convert the source documents into a string format that can be passed as context to the model. However, it's not clear how this function is implemented. If it's only returning the last document in the sourceDocuments array, that could explain why the model is only responding based on the last source document.

To ensure that all source documents are included in the model's context, you could modify the formatDocumentsAsString() function to concatenate all source documents into a single string. Here's an example of how you might do this:

function formatDocumentsAsString(documents) {
    return documents.map(doc => doc.content).join('\n');
}

This will create a single string that includes the content of all source documents, separated by newline characters.

As for including the source URLs in the model's response, you could modify the final step of the chain to append the source URLs to the model's output. Here's an example:

{
    result: prompt.pipe(model).pipe(new StringOutputParser()),
    sourceDocuments: (previousStepResult) => previousStepResult.sourceDocuments,
    finalResult: (previousStepResult) => {
        let result = previousStepResult.result;
        let sources = previousStepResult.sourceDocuments.map(doc => doc.url).join(', ');
        return `${result}\nSource(s): ${sources}`;
    },
}

This will append a line to the model's output that includes the source URLs of all source documents.

I hope this helps! If you have any further questions or if something is not clear, please let me know.

Sources

• examples/src/chains/question_answering_stuff.ts
• examples/src/chains/retrieval_qa_sources.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Simbaclaws]

@dosu-bot
formatDocumentAsString comes from langchain/util/document and isn't a function I'm able to change myself....

The current implementation of it looks like this:

export declare const formatDocumentsAsString: (documents: Document[]) => string;

I could of course create my own implementation.

I'm guessing that the result is coming from this line:
result: prompt.pipe(model).pipe(new StringOutputParser()),
And is basically just getting the last document in sourceDocuments for some reason.

[dosubot[bot]]

🤖

From the code you've shared, it seems like the RunnableSequence is correctly set up to pass the sourceDocuments through each step of the chain. However, the issue might be with how the sourceDocuments are being formatted and passed to the model.

The formatDocumentsAsString function is used to convert the sourceDocuments into a string that can be passed to the model. If this function is not correctly formatting the sourceDocuments, it could explain why the model is only responding based on the last document in the sourceDocuments array.

One possible solution could be to modify the formatDocumentsAsString function to ensure that it correctly formats all the sourceDocuments. However, as you've mentioned, this function is declared in the langchain/util/document module and you're not able to change it.

An alternative solution could be to create your own function to format the sourceDocuments. This function should take an array of Document objects and return a string that includes all the information from the sourceDocuments that you want the model to consider. You can then use this function instead of formatDocumentsAsString to format the sourceDocuments before passing them to the model.

Here's an example of how you could implement this function:

function formatAllDocumentsAsString(documents) {
    let formattedString = '';
    for (let doc of documents) {
        formattedString += doc.text + '\n';
        formattedString += 'Source: ' + doc.metadata.source + '\n';
    }
    return formattedString;
}

Then, in your RunnableSequence, you can replace the call to formatDocumentsAsString with a call to formatAllDocumentsAsString:

context: (previousStepResult) =>
    formatAllDocumentsAsString(previousStepResult.sourceDocuments),

This should ensure that all the sourceDocuments are included in the context that is passed to the model, and that the model's response includes the correct source URLs.

I hope this helps! If you have any further questions or if something is not clear, please let me know.

Sources

• examples/src/chains/multi_retrieval_qa.ts
• langchain-core/src/runnables/base.ts
• libs/langchain-community/src/vectorstores/faiss.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Simbaclaws]

I fixed my source code already, thanks for giving feedback bot :)