How do I handle image outputs from tools and feed it back to the LLM during an agent run?

[probably-harish]

I want to kind of replicate the usual method of sending a base64 encoded string to the llm using agents, as I want multiple images in the context. Is there already a way to do this or do I have to tamper with the source myself?

[seratch]

If your function tool returns base64-encoded data with sufficient description for LLM to tell how to use the data, the subsequent LLM calls can handle the data like when you send the same data as an input in user prompt.

[probably-harish]

example:

# to test image output from tool calls for agents

from agents import Agent, Runner, function_tool
import base64
import asyncio


IMAGE_PATH = 'data/sample.png'

@function_tool
def return_image_base64() -> str:
    with open(IMAGE_PATH, "rb") as image_file:
        # return f"data:image/png;base64,{base64.b64encode(image_file.read()).decode('utf-8')}"
        # return {
        #     "data_type": "data:image/png;base64",
        #     "data": base64.b64encode(image_file.read()).decode('utf-8')
        # }
        return "sample base64 string"


async def main():
    agent = Agent(
        name="Agent with image tools",
        instructions="""
        Mode 1: return "using mode 1; input received: {input}"
        Do NOT use the tool to return the image.

        Mode 2:
        You are an agent that I am using to test image tools. Use the tool `return_image_base64` to return a base64 encoded image. You can use the tool as many times as you want.
        You are to ALWAYS use the tool to return the image, even if the user request is not related to the image.
        The tool will return a base64 encoded image, as 
        return {
            "data_type": "data:image/png;base64",
            "data": base64.b64encode(image_file.read()).decode('utf-8')
        }
        NOTE: the return value from the tool should be sent not as string but as a dict with the key "data" and the value being the base64 encoded image.
        The subsequest request after the tool call should be in the following format and NOT as a string:
            input=[
            {
                "role": "user",
                "content": [
                    {"type": "input_text", "text": "describe the image in detail"},
                    {"type": "input_image", "image_url":  f"data:image/png;base64,{image_base64}"}
                ]
            }
        ]

        Return success message if the tool call is successful and we parsed as image and not as string. 
        """,
        tools=[return_image_base64],
        model="gpt-5-mini"
    )

    runner = await Runner.run(
        agent,
        input="Follow the instructions in your system prompt and use the tool that returns the image and respond as insturcted. Use mode 2 ",
        max_turns=2
    )

    print(runner.final_output)


if __name__ == "__main__":
    asyncio.run(main())

When i look at the logs, it looks like the return of the tool call is still stringified before the subsequent request to the responses endpoint, even with specific instructions? What am I missing @seratch ?

[probably-harish]

Relevant class method from agents/items.py

Tool call output item returned inside run implementation

        return [
            FunctionToolResult(
                tool=tool_run.function_tool,
                output=result,
                run_item=ToolCallOutputItem(
                    output=result,
                    raw_item=ItemHelpers.tool_call_output_item(tool_run.tool_call, str(result)),
                    agent=agent,
                ),
            )
            for tool_run, result in zip(tool_runs, results)
        ]

The type annotation for tool output is Any but seems like only objects on which str() can be applied are accepted?

[sebbe42]

All the digging I have been able to do says image urls can only be sent as role:user, not tool outputs.

Pydantic_ai does multimodal tool outputs by adding user messages trailing after a tool output. If you use the OpenAI api you can manually add these and it is interpreted correctly as tool outputs (I did add a bit of prompting though). I would rather use the agents sdk if this behavior could be easily implemented

[probably-harish]

@sebbe42 I understand that you were able to achieve this using Pydantic AI library + OpenAI API and NOT Agents SDK, right? Any other suggestions on how one might implement this behavior in Agents SDK itself? Or do you have any idea if the Google ADK has this feature built in as I wouldn't mind switching to the ADK

[sebbe42]

I meant that pydantic_AI does it out of the box. With a toolreturn string only message then trailing user messages with multimodal content. https://ai.pydantic.dev/tools/#function-tool-output

But you can do this natively with the OpenAI responses/chat api by just adding user messages after tool output message. But unless agents sdk can do this rather painlessly, you are giving up all tve good stuff.

Nothing wrong with pydantic_ai, it offers most of what agents sdk does, but as far as I can tell, no response storage or encrypted reasoning with responses api

[seratch]

Sorry, I totally forgot this SDK does not support this use case although I responded to a similar topic in the past: openai/openai-agents-js#285 As I mentioned in the PR comment, we don't have immediate plans to enable to send an additional user message item to enable send image data as we think it's not an intuitive design for agentic workflows. Our current recommendation is to store the image data in an accessible URL for LLM and relay the URL to LLM calls although I understand this may not work for your use case.

[sebbe42]

Thanks for the follow-up. Wondering if it’s possible to catch the tool call and inject the user messages manually, I’ll look into it when I have a chance. If you have the time for a pointer or two it would be very appreciated!

[probably-harish]

Seconding @sebbe42 on this

[sebbe42]

Ok so. Even if you cache images in the tool via the context wrapper and you catch the tool call via hooks, you cannot append any user messages to the runners items as they are not accessible. if you are really eager you'd have to modify the source code, but it'll be difficult.

Alternatives:

• use an llm call within the tool to evaluate the image and return that as part of the string. the tool can take as input any evaluation criteria from the main agent.
• grab the images from context after the run and automatically insert them before any next user messages, to ensure they end up in the history. you could go as far as prompting for 'after calling as the user to send you the images' with a certain syntax your application checks for, then automatically run the agent with the images without prompting the user for inputs.
• switch to pydantic_ai that natively supports the user message behavior, it works. Main drawback is they dont support carrying over reasoning from previous responses. (this was my motivation to switch to openai agents)

@seratch would you mind explaining how relaying the URL to the model will make the image end up in the conversation history? Also, since agents on the openai website can operate with images via code and reason about the outcomes (like cropping/zooming), agents are acting as if they have multimodal tool responses, at least on the website. So there is an internal workaround?

I have to say, given that models nowadays do tool calling within reasoning, its a substantial limitation that tool returns are not multimodal. its easily fixed when using the api and appending user messages, but while its not intuitive, it actually works and is simple.

[seratch]

Never mind my URL suggestion for your use case. There is no special support for URL inputs in this case. It can be just easier to extract tool output and pass the data to the subsequent LLM call if this works for your need like this one.

[seratch]

Regarding your use case, I understand this is a blocker for your desired app design, but calling LLM with image data within a tool seems to be the best fit with this SDK at the moment.

[sebbe42]

Thanks for the reply.