How can I use ConversationalRetrievalQAChain with images?

[Originn]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

I have a chatapp and I am testing the option of adding images and the bot answers using gpt-4-vision

I have this part of code which work fine, but I am missing the Template, chat history, that i usually get with ConversationalRetrievalQAChain, becasue if there is an image it will invoke streamingModel.


  const streamingModel = new ChatOpenAI({
    streaming: true,
    modelName: imageUrl ? IMAGE_MODEL_NAME : MODEL_NAME, // Select model based on presence of imageUrl
    temperature: TEMPRATURE,
    modelKwargs: {
      seed: 1
    },
    callbacks: [
      {
        handleLLMNewToken: (token) => {
          onTokenStream(token); // Forward the streamed token to the front-end
        },
      },
    ],
  });


      const chain = ConversationalRetrievalQAChain.fromLLM(
        streamingModel,
        customRetriever,
        {
          memory: chat_history,
          questionGeneratorChainOptions: {
            llm: nonStreamingModel,
          },
          qaTemplate: formattedPrompt,
          questionGeneratorTemplate: CONDENSE_PROMPT,
          returnSourceDocuments: true,
          verbose: true
        }
      );

      let responseText;

      if (imageUrl) {
        // Fetch and convert the image URL to base64
        const response = await fetch(imageUrl);
        const imageBuffer = await response.buffer();
        const imageMessage = new HumanMessage({
          content: [
            { type: "text", text: question },
            { type: 'image_url', image_url: { url: `data:image/jpeg;base64,${imageBuffer.toString('base64')}` } }
          ],
        });
    
        // Use the streamingModel to invoke with the image message
        responseText = await streamingModel.invoke([imageMessage]);
      } else {
        responseText = await chain.invoke({ question });
        const response = {
          text: responseText.text,
          sourceDocuments: responseText.sourceDocuments
      }
    
      
      };


but when I am changing it to invoke the chain, i get answers like "I'm sorry, but it seems there might have been an error with your question submission. It appears as "[object Object]" which does not provide me with any context or information to assist you."

```typescript
  const streamingModel = new ChatOpenAI({
    streaming: true,
    modelName: imageUrl ? IMAGE_MODEL_NAME : MODEL_NAME, // Select model based on presence of imageUrl
    temperature: TEMPRATURE,
    modelKwargs: {
      seed: 1
    },
    callbacks: [
      {
        handleLLMNewToken: (token) => {
          onTokenStream(token); // Forward the streamed token to the front-end
        },
      },
    ],
  });

      const chain = ConversationalRetrievalQAChain.fromLLM(
        streamingModel,
        customRetriever,
        {
          memory: chat_history,
          questionGeneratorChainOptions: {
            llm: nonStreamingModel,
          },
          qaTemplate: formattedPrompt,
          questionGeneratorTemplate: CONDENSE_PROMPT,
          returnSourceDocuments: true,
          verbose: true
        }
      );

      let responseText;

      if (imageUrl) {
        // Fetch and convert the image URL to base64
        const response = await fetch(imageUrl);
        const imageBuffer = await response.buffer();
        const imageMessage = new HumanMessage({
          content: [
            { type: "text", text: question },
            { type: 'image_url', image_url: { url: `data:image/jpeg;base64,${imageBuffer.toString('base64')}` } }
          ],
        });
    
        // Use the chain to invoke with the image message
        responseText = await chain.invoke({ question: imageMessage });
      } else {
        responseText = await chain.invoke({ question });
      }
    
      const response = {
        text: responseText.text,
        sourceDocuments: responseText.sourceDocuments
      };

What can I do in order to resolve the issue when calling the chain, on a message with image?

### Description

I am expecting that the ConversationalRetrievalQAChain will know how to handle a new HumanMessage with an image

const imageMessage = new HumanMessage({
content: [
{ type: "text", text: question },
{ type: 'image_url', image_url: { url: data:image/jpeg;base64,${imageBuffer.toString('base64')} } }
],
});

but I get response

I'm sorry, but it seems there might have been an error with your question submission. It appears as "[object Object]" which does not provide me with any context or information to assist you.

### System Info

[email protected] 

[dosubot[bot]]

The issue you're encountering with the ConversationalRetrievalQAChain not properly handling the HumanMessage object with an image seems to be related to how the message is being processed by the chain. Your approach to creating the HumanMessage with both text and image content is correct, as demonstrated by your code snippet. However, the error message indicating that the question submission appears as "[object Object]" suggests that the chain might not be interpreting the HumanMessage object correctly when it includes an image.

To address this, ensure the following:

• The image URL you're fetching is valid and accessible.
• The image data is correctly converted to a Base64 string.
• The ConversationalRetrievalQAChain is properly configured to handle HumanMessage objects with images.

If after checking these points you still face issues, providing more details about the error, such as the full error message and stack trace, could be crucial for further diagnosis. This additional context can help in pinpointing the exact cause of the problem and finding a solution.

Sources

• examples/src/models/chat/integration_anthropic_multimodal.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Originn]

"The ConversationalRetrievalQAChain is properly configured to handle HumanMessage objects with images."

That was my question...how to configure ConversationalRetrievalQAChain to handle HumanMessage objects with images.?