Why am I getting such poor response from Bedrock with VectorDBQAChain.fromLLM RAG?

[dmb0058]

I'm struggling to understand what's going on here, so just asking for insights or help :)

I've assembled the text of the GDPR regulations, generated embeddings using the Bedrock amazon.titan-embed-text-v1 model and saved these in a Faiss store.

I then do the normal RAG thing of using VectorDBQAChain.fromLLM() to vectorise my prompt using the same model and query the Bedrock amazon.titan-text-express-v1 model. I get the rather disappointing response "Sorry - this model is unable to respond to this request."

However, if I query OpenAI with the same Faiss vector store but using the OpenAI embedding model to vectorise my prompt, I get a sensible answer ("Article 25 of the regulation discusses data protection by design and by default. It requires that controllers implement appropriate technical and organizational measures to protect data subjects' rights and to meet the requirements of the regulation. It also states that a data protection impact assessment may be required in certain circumstances.").

The code is the same, so I don't understand why there's such variance - I'm clearly doing something wrong!

Any ideas very gratefully received :)

This is the Bedrock initialisation code:

async function queryBedrock(vs_filename, docs, question)
{
    const model = new Bedrock({
        model:"amazon.titan-text-express-v1",
        region: process.env.BEDROCK_REGION,
        credentials: {
            accessKeyId: process.env.AWS_ACCESS_KEY_ID,
            secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
        },
        temperature: 0.7,
        maxTokens: 2500
    });
    const embeddings = new BedrockEmbeddings({
        model: "amazon.titan-embed-text-v1",
        region: process.env.BEDROCK_REGION,
        credentials: {
            accessKeyId: process.env.AWS_ACCESS_KEY_ID,
            secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
        },
    });

    console.log('Query Amazon Bedrock:');
    await runQuery(model, embeddings, vs_filename, docs, question);
}

This is the equivalent OpenAI code:

async function queryOpenAi(vs_filename, docs, question)
{
    const model = new OpenAI({
        openAIApiKey: process.env.OPENAI_KEY,
        temperature: 0.7,
        max_tokens: 2500
    });
    const embeddings = new OpenAIEmbeddings({ openAIApiKey: process.env.OPENAI_KEY });

    console.log('Query OpenAI:');
    await runQuery(model, embeddings, vs_filename, docs, question);
}

And this is the runQuery() function that they both call to set up and call the chain:

async function runQuery(model, embeddings, vs_filename, docs, question)
{
    let vectorStore = await FaissStore.load(vs_filename, embeddings);

    const chain = VectorDBQAChain.fromLLM(
                        model,
                        vectorStore,
                        { k: 10, returnSourceDocuments: true }
                    );

    const res = await chain.call({
        input_documents: docs,
        query: question
    });

    console.log(question + "\n\n" + res.text );
}

[dosubot[bot]]

🤖

Hello again, @dmb0058! Nice to see you back. I hope you've been doing well. Let's take a look at this issue you're having.

Based on the provided context, it seems like the langchainjs codebase does support both the Bedrock and OpenAI models. However, the issue you're experiencing might be due to the specific models you're using or how they're being used.

In your Bedrock initialization code, you're using the amazon.titan-text-express-v1 model for the main model and amazon.titan-embed-text-v1 for embeddings. In the OpenAI code, you're not specifying a model, which means it's using the default model.

One possible reason for the variance in responses could be the difference in these models. Each model might have different capabilities and limitations, and they might handle the same input differently. It's also possible that the amazon.titan-text-express-v1 model doesn't support the specific type of request you're making.

Another potential issue could be the way you're using the models. In your runQuery() function, you're using the VectorDBQAChain.fromLLM() method to vectorize your prompt and query the model. It's possible that this method isn't compatible with the amazon.titan-text-express-v1 model, or that it requires a different approach.

Without more information, it's hard to say for sure what the issue is. I would recommend checking the documentation for the amazon.titan-text-express-v1 and amazon.titan-embed-text-v1 models to see if there are any specific requirements or limitations that could be causing this issue. You might also want to try using different models to see if that makes a difference.

If you're still having trouble, please provide more details about the issue, such as any error messages you're seeing and the exact steps you're taking to reproduce the issue. This will help me better understand what's going on and provide more targeted assistance.

Sources

• examples/src/models/llm/openai_userid.ts
• examples/src/models/llm/bedrock.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dmb0058]

@dosu-bot

I tried an alternative ... this gives the same sad response "Sorry - this model is unable to respond to this request.".
However if I type the same query into the AWS Bedrock Console I get a very good and informative result so the problem is not with the foundation model.

async function queryBedrock(vs_filename, docs, question)
{   
    const model = new Bedrock({
        model:"amazon.titan-text-express-v1",
        region: process.env.BEDROCK_REGION,
        credentials: {
            accessKeyId: process.env.AWS_ACCESS_KEY_ID,
            secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
        },
        temperature: 0.7,
        maxTokens: 2500
    });
    const embeddings = new BedrockEmbeddings({
        model: "amazon.titan-embed-text-v1",
        region: process.env.BEDROCK_REGION,
        credentials: {
            accessKeyId: process.env.AWS_ACCESS_KEY_ID,
            secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
        },
    });
    
    let vectorStore = await FaissStore.load(vs_filename, embeddings);
    const retriever = vectorStore.asRetriever();
    const prompt = PromptTemplate.fromTemplate(`Answer the question based only on the following context: {context} Question: {question}`);
    const chain = RunnableSequence.from([
                    {   
                        context: retriever.pipe(formatDocumentsAsString),
                        question: new RunnablePassthrough(),
                    },
                    prompt,
                    model,
                    new StringOutputParser(),
                ]);
    
    const result = await chain.invoke(question);
    console.log(result);
}

[dosubot[bot]]

🤖

Based on the code you've provided, it seems like you're not using the VectorDBQAChain.fromLLM() method, but rather a custom sequence of operations. However, I understand that you're experiencing an issue with the Bedrock model not being able to respond to your request.

The error message "Sorry - this model is unable to respond to this request" is typically returned by the underlying language model when it is unable to generate a response for the given input. This could be due to a variety of reasons, such as the input not being in a format the model expects, the model not being properly trained, or the model not being able to generate a response within the specified token limit.

Here are a few things you can check:

1. Input Format: Ensure that the input to the model is in the format it expects. For the Bedrock model, this would typically be a string of text. If you're passing in an object or some other data type, the model might not be able to handle it.

2. Model Training: Check that the Bedrock model is properly trained. If the model hasn't been trained on the type of data you're querying it with, it might not be able to generate a sensible response.

3. Token Limit: The maxTokens parameter specifies the maximum number of tokens the model should generate for its response. If this value is too low, the model might not be able to generate a complete response. Try increasing this value to see if it resolves the issue.

4. Temperature: The temperature parameter controls the randomness of the model's output. A higher value will result in more random output, while a lower value will result in more deterministic output. If the temperature is too high, the model might generate nonsensical or unrelated responses. Try lowering this value to see if it improves the quality of the responses.

If none of these suggestions solve your problem, it would be helpful to know more about the exact input you're passing to the model and the context in which you're using it. This could help identify any potential issues with the way the model is being used.

For more information on how to use the Bedrock model with the LangChainJS framework, you can refer to the langchain/src/chains/vector_db_qa.ts file in the LangChainJS repository. This file contains the implementation of the VectorDBQAChain class, which might be helpful for understanding how to use the Bedrock model in this context.

I hope this helps! If you have any other questions or need further clarification, please don't hesitate to ask.

Sources

• langchain/src/chains/vector_db_qa.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dmb0058]

@dosu-bot

The result is not reliable anyway. If I directly invoke the model I usually get a sensible result but even that sometimes gives "Sorry - this model is unable to respond to this request.":

$ node index-4.js
Query Amazon Bedrock:
question: 'what does article 25 of the gdpr say?'
Direct invocation: Article 25 of the GDPR states that the data subject has the right to request the erasure of his or her personal data from the controller without undue delay. The controller shall have the right to determine the time limits for erasure.
Individual steps: Sorry - this model is unable to respond to this request.
Use VectorDBAQChain: Sorry - this model is unable to respond to this request.

$ node index-4.js
Query Amazon Bedrock:
question: 'what does article 25 of the gdpr say?'
Direct invocation: Sorry - this model is unable to respond to this request.
Individual steps: Sorry - this model is unable to respond to this request.
Use VectorDBAQChain: Sorry - this model is unable to respond to this request.

How can I debug this code? I don't know what's flakey here, langchain or the FM:

async function queryBedrock(vs_filename, docs, question)
{   
    const model = new Bedrock({
        model:"amazon.titan-text-express-v1",
        region: process.env.BEDROCK_REGION,
        credentials: {
            accessKeyId: process.env.AWS_ACCESS_KEY_ID,
            secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
        },
        temperature: 0.7,
        maxTokens: 2500
    });

    /////////////////////////////
    // invoke model directly
    /////////////////////////////

    try {
        let result = await model.invoke(question);
        console.log(`Direct invocation: ${result}`);
    }
    catch (e){
        console.log(e);
    }

    /////////////////////////////
    // use individual steps
    /////////////////////////////

    const embeddings = new BedrockEmbeddings({
        model: "amazon.titan-embed-text-v1",
        region: process.env.BEDROCK_REGION,
        credentials: {
            accessKeyId: process.env.AWS_ACCESS_KEY_ID,
            secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
        },
    });
    
    let vectorStore = await FaissStore.load(vs_filename, embeddings);
    const retriever = vectorStore.asRetriever();
    const prompt = PromptTemplate.fromTemplate(`Answer the question based only on the following context: {context} Question: {question}`);
    const indiv_chain = RunnableSequence.from([
                    {   
                        context: retriever.pipe(formatDocumentsAsString),
                        question: new RunnablePassthrough(),
                    },
                    prompt,
                    model,
                    new StringOutputParser(),
                ]);
    
    const result = await indiv_chain.invoke(question);
    console.log(`Individual steps: ${result}`);

    /////////////////////////////
    // use VectorDBQAChain
    /////////////////////////////

    const dbqa_chain = VectorDBQAChain.fromLLM(
                        model,
                        vectorStore,
                        { k: 10, returnSourceDocuments: true }
                    );

    const res = await dbqa_chain.call({
        input_documents: docs,
        query: question
    });
    console.log(`Use VectorDBAQChain: ${result}`);
}

[dosubot[bot]]

🤖

To debug the inconsistency in responses from the amazon.titan-text-express-v1 model in the LangChainJS framework, you can use the callbacks option in the ChatOpenAI constructor. This allows you to log the input and output of each call to the model, as well as any errors that occur.

Here's how you can modify your code to debug the amazon.titan-text-express-v1 model:

import { HumanMessage, LLMResult } from "langchain/schema";
import { ChatOpenAI } from "langchain/chat_models/openai";
import { Serialized } from "langchain/load/serializable";

// We can pass in a list of CallbackHandlers to the LLM constructor to get callbacks for various events.
const model = new ChatOpenAI({
  callbacks: [
    {
      handleLLMStart: async (llm: Serialized, prompts: string[]) => {
        console.log('LLM Start');
        console.log(JSON.stringify(llm, null, 2));
        console.log(JSON.stringify(prompts, null, 2));
      },
      handleLLMEnd: async (output: LLMResult) => {
        console.log('LLM End');
        console.log(JSON.stringify(output, null, 2));
      },
      handleLLMError: async (err: Error) => {
        console.log('LLM Error');
        console.error(err);
      },
    },
  ],
});

await model.call([
  new HumanMessage(
    "What is a good name for a company that makes colorful socks?"
  ),
]);

This will give you detailed logs of the input, output, and any errors for each call to the model. You can then use these logs to identify any patterns in the inconsistent responses, which can help you debug the issue.

You can find more information about this in the examples/src/models/chat/chat_debugging.ts file in the LangChainJS repository.

As for the expected behavior of the amazon.titan-text-express-v1 model when invoked through the LangChainJS framework, I wasn't able to find specific information within the repository. The behavior might depend on various factors such as the input data, the model configuration, and the specific use case. If the issue persists, it might be helpful to reach out to the maintainers of the amazon.titan-text-express-v1 model for more specific guidance.

Sources

• examples/src/models/chat/chat_debugging.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dmb0058]

@dosu-bot Do you know of an active forum for active langchain/Bedrock developers?

…

On Tue, 5 Dec 2023, 14:49 dosubot[bot], ***@***.***> wrote: *🤖* To debug the inconsistency in responses from the amazon.titan-text-express-v1 model in the LangChainJS framework, you can use the callbacks option in the ChatOpenAI constructor. This allows you to log the input and output of each call to the model, as well as any errors that occur. Here's how you can modify your code to debug the amazon.titan-text-express-v1 model: import { HumanMessage, LLMResult } from "langchain/schema";import { ChatOpenAI } from "langchain/chat_models/openai";import { Serialized } from "langchain/load/serializable"; // We can pass in a list of CallbackHandlers to the LLM constructor to get callbacks for various events.const model = new ChatOpenAI({ callbacks: [ { handleLLMStart: async (llm: Serialized, prompts: string[]) => { console.log('LLM Start'); console.log(JSON.stringify(llm, null, 2)); console.log(JSON.stringify(prompts, null, 2)); }, handleLLMEnd: async (output: LLMResult) => { console.log('LLM End'); console.log(JSON.stringify(output, null, 2)); }, handleLLMError: async (err: Error) => { console.log('LLM Error'); console.error(err); }, }, ],}); await model.call([ new HumanMessage( "What is a good name for a company that makes colorful socks?" ),]); This will give you detailed logs of the input, output, and any errors for each call to the model. You can then use these logs to identify any patterns in the inconsistent responses, which can help you debug the issue. You can find more information about this in the examples/src/models/chat/chat_debugging.ts <https://github.com/langchain-ai/langchainjs/blob/main/examples/src/models/chat/chat_debugging.ts> file in the LangChainJS repository. As for the expected behavior of the amazon.titan-text-express-v1 model when invoked through the LangChainJS framework, I wasn't able to find specific information within the repository. The behavior might depend on various factors such as the input data, the model configuration, and the specific use case. If the issue persists, it might be helpful to reach out to the maintainers of the amazon.titan-text-express-v1 model for more specific guidance. Sources - examples/src/models/chat/chat_debugging.ts <https://github.com/langchain-ai/langchainjs/blob/main/examples/src/models/chat/chat_debugging.ts> ------------------------------ This response is meant to be useful and save you time. It is *not meant to be a precise solution*, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot <https://github.com/dosu-bot>. — Reply to this email directly, view it on GitHub <#3525 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDXCQPBKL2SMSVXO7CKZW3YH4YAZAVCNFSM6AAAAABAGLTJXKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TONRVGUZTE> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[dosubot[bot]]

🤖

Based on the code you've shared, it seems like you're using the amazon.titan-text-express-v1 model from Bedrock in three different ways: direct invocation, individual steps, and through the VectorDBQAChain. The inconsistency in responses could be due to a variety of factors, including the specific inputs you're using, the configuration of the model, or even potential issues with the model itself.

However, without more specific information about the inputs you're using and the exact circumstances under which the error occurs, it's difficult to pinpoint the exact cause of the issue.

As for debugging, the callbacks option in the ChatOpenAI constructor can indeed be useful for understanding the internal workings of the model. However, it might not provide enough information to diagnose this specific issue. You might want to consider adding additional logging or error handling code to your application to capture more detailed information about the state of the application when the error occurs.

Regarding your question about an active forum for LangChain and Bedrock developers, I'm afraid I don't have any specific recommendations. However, you might find it helpful to reach out to the maintainers of the LangChainJS and Bedrock projects directly. They might be able to provide more specific guidance or point you towards resources that can help you troubleshoot your issue.

I hope this helps, and please don't hesitate to reach out if you have any further questions or if there's anything else I can do to assist you.

Sources

• examples/src/guides/fallbacks/better_model.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dmb0058]

@dosu-bot "The inconsistency in responses could be due to a variety of factors, including the specific inputs you're using, the configuration of the model, or even potential issues with the model itself." Makes sense to me, but the input and configuration is the same in each case, and I think it's unlikely that the model is broken. I suspect it's a different issue to any of these. I'll try the project teams.

…

On Tue, 5 Dec 2023, 19:01 dosubot[bot], ***@***.***> wrote: *🤖* Based on the code you've shared, it seems like you're using the amazon.titan-text-express-v1 model from Bedrock in three different ways: direct invocation, individual steps, and through the VectorDBQAChain. The inconsistency in responses could be due to a variety of factors, including the specific inputs you're using, the configuration of the model, or even potential issues with the model itself. However, without more specific information about the inputs you're using and the exact circumstances under which the error occurs, it's difficult to pinpoint the exact cause of the issue. As for debugging, the callbacks option in the ChatOpenAI constructor can indeed be useful for understanding the internal workings of the model. However, it might not provide enough information to diagnose this specific issue. You might want to consider adding additional logging or error handling code to your application to capture more detailed information about the state of the application when the error occurs. Regarding your question about an active forum for LangChain and Bedrock developers, I'm afraid I don't have any specific recommendations. However, you might find it helpful to reach out to the maintainers of the LangChainJS and Bedrock projects directly. They might be able to provide more specific guidance or point you towards resources that can help you troubleshoot your issue. I hope this helps, and please don't hesitate to reach out if you have any further questions or if there's anything else I can do to assist you. Sources - examples/src/guides/fallbacks/better_model.ts <https://github.com/langchain-ai/langchainjs/blob/main/examples/src/guides/fallbacks/better_model.ts> ------------------------------ This response is meant to be useful and save you time. It is *not meant to be a precise solution*, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot <https://github.com/dosu-bot>. — Reply to this email directly, view it on GitHub <#3525 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDXCQJRQ2R7DKT3IRKEKJLYH5VQXAVCNFSM6AAAAABAGLTJXKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TONRXHE3TA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[dosubot[bot]]

🤖

It sounds like you've been encountering some inconsistencies in the responses from the language model you're using. I understand your concern about the input and configuration being the same in each case, and it's reasonable to suspect that the issue might be different from the factors I mentioned.

Reaching out to the project teams is a good step, as they might have insights into the specific behavior of the model and any potential issues that could be causing the inconsistencies.

If you need further assistance or have any other questions, feel free to ask. I'm here to help!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Sonali-Behera-TRT]

Hey @dmb0058, Did you find any solution to this ?

[dmb0058]

Hi, No, unfortunately I never solved it, but got diverted onto other projects. I'd like to be able to use Bedrock as an alternative so hopefully I'll get time at some point to investigate further. Are you having a similar problem?

…

On Fri, 19 Apr 2024, 11:36 Sonali-Behera-TRT, ***@***.***> wrote: Hey @dmb0058 <https://github.com/dmb0058>, Did you find any solution to this ? — Reply to this email directly, view it on GitHub <#3525 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDXCQMHZP326HXCLYS6XMDY6DXTTAVCNFSM6AAAAABAGLTJXKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCNRVGE3DK> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[Sonali-Behera-TRT]

Yes @dmb0058