How to let a tool return a picture?

[LetianYuan]

I am using a vision LLM model.

I know how to directly send a picture to it using graph.invoke, graph.stream, or raw HTTP request.

But, my problem is, how to let a tool return a picture?

For example, when the LLM calls "get_screenshot" tool, I want to provide a screenshot of a website to it.

from langchain_core.tools import tool

def do_screenshoot_and_save_it_to_local_disk():
    # Pretend that this function will take a screenshot and return the path of the image.
    return "/path/to/pic.png"

@tool
def get_screenshot():
    picture_path = do_screenshoot_and_save_it_to_local_disk()
    return ??? # What should I return?

How to implement it? Or, is it impossible to let a tool return a picture when using langgraph?

Or, when a function is decorated with @tool, what types of return values are allowed? Obviously, returning a str cannot provide a picture for the vision LLM.

[dionysis11]

Hello! I Did a quick research and I found that tools can either return a string or a list of dictionaries. In the second option you can specify which type you want to pass which should be one of these: ("image", "image_url", "text", "json"), where as for the image you should pass a base64 object and the return should look something like:

return [
        {"type": "image", "image": {"data": encoded_string, "mime_type": "image/jpeg"}}
    ]

I would though recommend to be careful not to have many base64 in memory at once since it can easily result in a MemoryError!
Hope I was helpful, please correct me if I'm wrong!

[LetianYuan]

@dionysis11 Thank you very much. This problem has been troubling me for a week.
Here's the full code. It works well.

import base64

from langchain_core.tools import tool


def image_to_base64(image_path: str) -> str:
    with open(image_path, "rb") as image_file:
        encoded_string = base64.b64encode(image_file.read())
        return encoded_string.decode('utf-8')


def do_screenshot_and_save_it_to_local_disk():
    # Pretend that this function will take a screenshot and return the path of the image.
    return "/path/to/pic.png"


@tool
def get_screenshot():
    picture_path = do_screenshot_and_save_it_to_local_disk()
    return [
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_to_base64(picture_path)}"}}
    ]

[Harshbansal8705]

It gives me the following error:

groq.BadRequestError: Error code: 400 - {'error': {'message': "'messages.3' : for 'role:tool' the following must be satisfied[('messages.3.content' : value must be a string)]", 'type': 'invalid_request_error'}}

Here's the code for my tool:

from langchain_core.tools import tool
from PIL import ImageGrab

@tool
def get_screenshot():
    """
    Get a screenshot of the current screen
    """
    logger.info("[get_screenshot] Taking screenshot...")
    screenshot = ImageGrab.grab()
    screenshot.save("screenshot.png")
    with open("screenshot.png", "rb") as image_file:
        image_data = base64.b64encode(image_file.read()).decode("utf-8")

    return [
        {
            "type": "image_url",
            "image_url": {"url": f"data:image/png;base64,{image_data}"},
        }
    ]

[polavishnu4444]

Did this work for OpenAI used ToolMessage response?