Documentation on Images is Misleading

[maximepeabody]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

When working with local images, the BinaryContent type is misleading. Docs make it seem like this abstracts away file upload with the model provider, but instead it seems to pass the raw data as string in the prompt, which quickly consumes the entire token context window.

Example Code

Python, Pydantic AI & LLM client version

pypdantic-ai 1.0.15
gemini client (tried openai as well)
python 3.11.9

[DouweM]

@maximepeabody With BinaryContent, we use the model API's native method of receiving binary data, which is typically to send bytes or base64-based data URI.

What parts of the docs are misleading about this?

[maximepeabody]

Ah, I think I assumed the wrong issue. I'm digging deeper into my logs, and it looks like the issue is that I'm returning BinaryContent images in one of my functions inside of a tuple. Pydantic-AI is then not handling this correctly i believe

If a BinaryContent image gets returned by a function alone would it work? Or is this a limitation?

[DouweM]

@maximepeabody Ah yeah, returned directly or in a list would work, we haven't added special handling for tuples.