Multimodal chat with GPT-4o - Including history and variable images: How-to?

[skye0402]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# Code is below

Description

I'm trying to pass a user question and images to GPT-4o. The amount of images is anything between zero to 'n'. The user can decide.
This works without problems if I apply this:

def get_multimodal_prompt(input_data: dict)->dict:
    """ 
        Create the prompt for OpenAI GPT-4o
    """
    text = input_data["text"]
    images = input_data["images"]
    content_parts = []
    for image in images:
        image_part = {
            "type": "image_url",
            "image_url": {
                "url": image
            }
        }
        content_parts.append(image_part)    
    text_part = {"type": "text", "text": text}    
    content_parts.append(text_part)
    return [HumanMessage(content=content_parts)]

and

if model_info["hasImage"]: # Multimodal call            
            state["chain"] = (
                get_multimodal_prompt 
                | state["model"]
                | StrOutputParser()
            )        

finally

                        for response in state["chain"].stream({"text": input_from_chat, "images": image_base64}):
                            history[-1][1] += response
                            logging.debug(history[-1][1])
                            yield history

However, I want it to work with history and a proper Chatprompt. I'm not able to bring it to life.
I mean this (which is for pure text):

state["chat_prompt"] = ChatPromptTemplate.from_messages(
                [
                    SystemMessage(content=system_role),
                    MessagesPlaceholder(variable_name="history"),
                    HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['input'], template="{input}"))
                ],
            )
            state["chain"] = (
                {
                    "history": RunnableLambda(state["memory"].load_memory_variables) | itemgetter("history"), 
                    "input": RunnablePassthrough()
                }
                | state["chat_prompt"]
                | state["model"]
                | StrOutputParser()
            ) 

Any suggestions? I can't make it work.

System Info

langchain==0.2.7
langchain-anthropic==0.1.15
langchain-cohere==0.1.8
langchain-community==0.2.7
langchain-core==0.2.12
langchain-google-genai==1.0.3
langchain-openai==0.1.14
langchain-text-splitters==0.2.2

[dosubot[bot]]

To integrate history and a proper ChatPrompt for a multimodal setup in your existing code, you can follow these steps:

1. Setup Dependencies:
   Ensure you have the required packages installed:

   %pip install --upgrade --quiet langchain langchain-openai faiss-cpu tiktoken

2. Set Environment Variables:

   import getpass
   import os
   
   if not os.environ.get("OPENAI_API_KEY"):
       os.environ["OPENAI_API_KEY"] = getpass.getpass()

3. Initialize LLM:

   from langchain_openai import ChatOpenAI
   
   llm = ChatOpenAI(model="gpt-4o", temperature=0)

4. Create Retriever:

   from langchain_community.vectorstores import FAISS
   from langchain_openai import OpenAIEmbeddings
   
   vectorstore = FAISS.from_texts(
       ["harrison worked at kensho"], embedding=OpenAIEmbeddings()
   )
   retriever = vectorstore.as_retriever()

5. Create ChatPrompt:

   from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder, SystemMessage, HumanMessagePromptTemplate, PromptTemplate
   
   system_role = "You are a helpful assistant."
   template = """Answer the question based only on the following context:
   {context}
   
   Question: {question}
   """
   prompt = ChatPromptTemplate.from_template(template)

6. Create History-Aware Retriever:

   from langchain_core.messages import AIMessage, HumanMessage, get_buffer_string
   from langchain_core.prompts import format_document, PromptTemplate
   from langchain_core.runnables import RunnableParallel, RunnablePassthrough, RunnableLambda
   from operator import itemgetter
   
   _template = """Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.
   
   Chat History:
   {chat_history}
   Follow Up Input: {question}
   Standalone question:"""
   CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(_template)
   
   DEFAULT_DOCUMENT_PROMPT = PromptTemplate.from_template(template="{page_content}")
   
   def _combine_documents(docs, document_prompt=DEFAULT_DOCUMENT_PROMPT, document_separator="\n\n"):
       doc_strings = [format_document(doc, document_prompt) for doc in docs]
       return document_separator.join(doc_strings)
   
   _inputs = RunnableParallel(
       standalone_question=RunnablePassthrough.assign(
           chat_history=lambda x: get_buffer_string(x["chat_history"])
       )
       | CONDENSE_QUESTION_PROMPT
       | ChatOpenAI(temperature=0)
       | StrOutputParser(),
   )
   _context = {
       "context": itemgetter("standalone_question") | retriever | _combine_documents,
       "question": lambda x: x["standalone_question"],
   }
   conversational_qa_chain = _inputs | _context | prompt | ChatOpenAI()

7. Manage Chat History:

   from langchain.memory import ConversationBufferMemory
   
   memory = ConversationBufferMemory(
       return_messages=True, output_key="answer", input_key="question"
   )
   
   loaded_memory = RunnablePassthrough.assign(
       chat_history=RunnableLambda(memory.load_memory_variables) | itemgetter("history"),
   )
   standalone_question = {
       "standalone_question": {
           "question": lambda x: x["question"],
           "chat_history": lambda x: get_buffer_string(x["chat_history"]),
       }
       | CONDENSE_QUESTION_PROMPT
       | ChatOpenAI(temperature=0)
       | StrOutputParser(),
   }
   retrieved_documents = {
       "docs": itemgetter("standalone_question") | retriever,
       "question": lambda x: x["standalone_question"],
   }
   final_inputs = {
       "context": lambda x: _combine_documents(x["docs"]),
       "question": itemgetter("question"),
   }
   answer = {
       "answer": final_inputs | prompt | ChatOpenAI(),
       "docs": itemgetter("docs"),
   }
   final_chain = loaded_memory | standalone_question | retrieved_documents | answer

8. Invoke the Chain:

   inputs = {"question": "where did harrison work?"}
   result = final_chain.invoke(inputs)
   print(result)
   
   memory.save_context(inputs, {"answer": result["answer"].content})
   
   inputs = {"question": "but where did he really work?"}
   result = final_chain.invoke(inputs)
   print(result)

This setup will allow you to integrate chat history and a proper ChatPrompt for a multimodal setup in your existing code. For more advanced topics like managing large message history and streaming responses, refer to the LangChain documentation ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[skye0402]

This is not helpful - it explains a RAG scenario. I want to send images to Openai GPT-4o using a chat history.

[dosubot[bot]]

To integrate chat history and a proper ChatPrompt for sending text and images to OpenAI GPT-4o in a multimodal setup, you can modify your current setup as follows:

1. Import necessary libraries and encode the image:

   import base64
   import httpx
   
   image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
   image_data = base64.b64encode(httpx.get(image_url).content).decode("utf-8")

2. Initialize the model:

   from langchain_core.prompts import ChatPromptTemplate
   from langchain_openai import ChatOpenAI
   
   model = ChatOpenAI(model="gpt-4o")

3. Create a ChatPromptTemplate with chat history and multimodal inputs:

   chat_history = [
       ("system", "You are a helpful assistant."),
       ("user", "Can you describe the image I am sending?"),
       ("assistant", "Sure, please provide the image."),
   ]
   
   prompt = ChatPromptTemplate.from_messages(
       chat_history + [
           (
               "user",
               [
                   {
                       "type": "image_url",
                       "image_url": {"url": f"data:image/jpeg;base64,{image_data}"},
                   }
               ],
           ),
       ]
   )

4. Create the chain and invoke it with the image data:

   chain = prompt | model
   response = chain.invoke({"image_data": image_data})
   print(response.content)

This setup includes a chat history and integrates the image data into the prompt, allowing you to send both text and images to the OpenAI GPT-4o model in a multimodal setup.

Additionally, you can use the RunnableLambda to format the inputs and handle the multimodal data more effectively. Here is an example of how to use RunnableLambda for a multimodal RAG setup:

from langchain_core.runnables import RunnableLambda, RunnablePassthrough
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
from langchain_core.output_parsers import StrOutputParser

def prompt_func(data_dict):
    formatted_texts = "\n".join(data_dict["text_context"]["texts"])
    messages = []

    if data_dict["image_context"]["images"]:
        image_message = {
            "type": "image_url",
            "image_url": {
                "url": f"data:image/jpeg;base64,{data_dict['image_context']['images'][0]}"
            },
        }
        messages.append(image_message)

    text_message = {
        "type": "text",
        "text": (
            "As an expert art critic and historian, your task is to analyze and interpret images, "
            "considering their historical and cultural significance. Alongside the images, you will be "
            "provided with related text to offer context. Both will be retrieved from a vectorstore based "
            "on user-input keywords. Please use your extensive knowledge and analytical skills to provide a "
            "comprehensive summary that includes:\n"
            "- A detailed description of the visual elements in the image.\n"
            "- The historical and cultural context of the image.\n"
            "- An interpretation of the image's symbolism and meaning.\n"
            "- Connections between the image and the related text.\n\n"
            f"User-provided keywords: {data_dict['question']}\n\n"
            "Text and / or tables:\n"
            f"{formatted_texts}"
        ),
    }
    messages.append(text_message)

    return [HumanMessage(content=messages)]

model = ChatOpenAI(temperature=0, model="gpt-4-vision-preview", max_tokens=1024)
chain = (
    {
        "text_context": text_retriever | RunnableLambda(split_image_text_types),
        "image_context": image_retriever | RunnableLambda(split_image_text_types),
        "question": RunnablePassthrough(),
    }
    | RunnableLambda(prompt_func)
    | model
    | StrOutputParser()
)

This approach ensures that both text and image data are properly formatted and sent to the GPT-4o model for a comprehensive multimodal response ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[skye0402]

@dosu Yes, you example is for one image, how to make it for a variable amount of images? And how to incorporate chat history?

[dosubot[bot]]

To handle a variable number of images and incorporate chat history in your multimodal setup with OpenAI GPT-4o, you can modify your current setup as follows:

1. Define the function to create the prompt with variable images and chat history:

   def get_multimodal_prompt(input_data: dict) -> dict:
       """
       Create the prompt for OpenAI GPT-4o with variable images and chat history.
       """
       text = input_data["text"]
       images = input_data["images"]
       chat_history = input_data.get("chat_history", [])
       
       content_parts = []
       
       # Add chat history to the prompt
       for chat in chat_history:
           content_parts.append({"type": "text", "text": chat})
       
       # Add images to the prompt
       for image in images:
           image_part = {
               "type": "image_url",
               "image_url": {"url": image}
           }
           content_parts.append(image_part)
       
       # Add the main text to the prompt
       text_part = {"type": "text", "text": text}
       content_parts.append(text_part)
       
       return [HumanMessage(content=content_parts)]

2. Set up the chain to include the chat history and handle multiple images:

   from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder, SystemMessage, HumanMessagePromptTemplate, PromptTemplate
   from langchain_openai import ChatOpenAI
   from langchain_core.runnables import RunnableLambda, RunnablePassthrough
   from langchain_core.output_parsers import StrOutputParser
   from operator import itemgetter
   
   # Initialize the model
   model = ChatOpenAI(model="gpt-4o")
   
   # Define the chat prompt template
   system_role = "You are a helpful assistant."
   chat_prompt = ChatPromptTemplate.from_messages(
       [
           SystemMessage(content=system_role),
           MessagesPlaceholder(variable_name="history"),
           HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['input'], template="{input}"))
       ]
   )
   
   # Define the chain
   state = {
       "memory": ConversationBufferMemory(return_messages=True, output_key="answer", input_key="question"),
       "model": model,
       "chat_prompt": chat_prompt
   }
   
   state["chain"] = (
       {
           "history": RunnableLambda(state["memory"].load_memory_variables) | itemgetter("history"),
           "input": RunnablePassthrough()
       }
       | state["chat_prompt"]
       | state["model"]
       | StrOutputParser()
   )

3. Invoke the chain with the input data including chat history and images:

   input_data = {
       "text": "Can you describe the images?",
       "images": ["data:image/jpeg;base64,...", "data:image/jpeg;base64,..."],  # Add your base64 encoded images here
       "chat_history": [
           "User: Hi, can you help me with something?",
           "Assistant: Sure, what do you need help with?"
       ]
   }
   
   # Create the prompt with the input data
   prompt = get_multimodal_prompt(input_data)
   
   # Invoke the chain
   response = state["chain"].invoke({"input": prompt})
   print(response.content)

This setup ensures that both the chat history and a variable number of images are included in the prompt sent to the OpenAI GPT-4o model. The get_multimodal_prompt function dynamically handles the number of images and incorporates the chat history into the prompt ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[yunhzou]

Encountered the same problem. Waiting for a fix