Can not send BinaryContent to "gpt-4.1-mini" on AzureOpenAI

[sugawarayss]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

i facing issue about it is not working BinaryContent with AzureOpenAI "gpt-4.1-mini" model .

Agent.run have user_prompt string(written in japanese) and 9 image files (base64 encoded content).
gpt-4.1-mini model is not include BinaryContent in request_tokens like following Usage information.

does pydantic-ai not support "gpt-4.1-mini"? or my code is not something bad???

When i use "gpt-4o-mini" model

Usage(
    requests=1,
    request_tokens=78480,
    response_tokens=124,
    total_tokens=78604,
    details={'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0, 'cached_tokens': 0}
)

When i use "gpt-4.1-mini" model

Usage(
    requests=1,
    request_tokens=2889,  # it is not include BinaryContent????
    response_tokens=17,
    total_tokens=2906,
    details={'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0, 'cached_tokens': 0}
)

Example Code

model = OpenAIModel(
    model_name=MY_AZURE_DEPLOYMENT_NAME,  # deploy model:"gpt-4.1-mini" with model_ver: "2025-04-14"
    provider=AzureProvider(
        azure_endpoint=MY_AZURE_ENDPOINT,
        api_key=MY_AZURE_API_KEY,
        api_version=MY_AZURE_API_VERSION,  # "2024-10-21"
)
my_agent = Agent(
    model=model,
    system_prompt=MY_SYSTEM_PROMPT,
    output_type=MyResultPydanticModel,
)

binary_contents: list[BinaryContent] = [BinaryContent(data=file.content, media_type=str(file.mime_type)) for file in params.files]
user_prompt = [
    USER_PROMPT_TEMPLATE,
    *binary_contents,
]
agent_run: AgentRunResult[Any] = await my_agent.run(user_prompt=user_prompt)

print(f"{agent_run.usage()}")

Python, Pydantic AI & LLM client version

Python: 3.13.1
Pydantic: 2.11.3
Pydantic-ai: 0.6.2
Model: gpt-4.1-mini(AzureOpenAI)

[DouweM]

@sugawarayss You can use https://ai.pydantic.dev/logfire to verify what is actually being sent to and received from the API. You can also likely see from the agent output whether the binary content was taken into account or not. We do support BinaryContent with all OpenAI models, so I expect this to be a bug in the API where it's not taking the binary content into account. But let me know if Logfire or the agent's output suggests that the BinaryContent is not being sent at all.

[sugawarayss]

@DouweM
thank you for your reply. i will try to use logfire!

[sugawarayss]

@DouweM
While waiting for Logfire to be used, I tried capturing messages with capture_run_messages().
The value of BinaryContent.data is long, so it has been omitted (and mask system/user prompt) in the following code block (although the value was actually set).

BinaryContent was set on ModelRequest, but it didn't seem to be included in request_tokens.

[
    ModelRequest(
        parts=[
            SystemPromptPart(
                content='MY SYSTEM PROMPT STRING',
                timestamp=datetime.datetime(2025, 8, 12, 4, 33, 19, 92080, tzinfo=datetime.timezone.utc),
                dynamic_ref=None,
                part_kind='system-prompt',
            ),
            SystemPromptPart(
                content='MY SYSTEM PROMPT STRING',
                timestamp=datetime.datetime(2025, 8, 12, 4, 33, 19, 92327, tzinfo=datetime.timezone.utc),
                dynamic_ref=None,
                part_kind='system-prompt',
            ),
            UserPromptPart(
                content=[
                    'MY USER PROMPT STRING',
                    BinaryContent(
                        data=b'ommit. it too long',
                        media_type='image/png',
                        kind='binary',
                    ),
                    BinaryContent(
                        data=b'ommit. it too long',
                        media_type='image/png',
                        kind='binary',
                    ),
                    BinaryContent(
                        data=b'ommit. it too long',
                        media_type='image/png',
                        kind='binary',
                    ),
                    BinaryContent(
                        data=b'ommit. it too long',
                        media_type='image/png',
                        kind='binary',
                    ),
                    BinaryContent(
                        data=b'ommit. it too long',
                        media_type='image/png',
                        kind='binary',
                    ),
                    BinaryContent(
                        data=b'ommit. it too long',
                        media_type='image/png',
                        kind='binary',
                    ),
                    BinaryContent(
                        data=b'ommit. it too long',
                        media_type='image/png',
                        kind='binary',
                    ),
                    BinaryContent(
                        data=b'ommit. it too long',
                        media_type='image/png',
                        kind='binary',
                    ),
                    BinaryContent(
                        data=b'ommit. it too long',
                        media_type='image/png',
                        kind='binary',
                    )
                ],
                timestamp=datetime.datetime(2025, 8, 12, 4, 33, 19, 92331, tzinfo=datetime.timezone.utc),
                part_kind='user-prompt',
            )
        ],
        instructions=None,
        kind='request',
    ),
    ModelResponse(
        parts=[
            TextPart(content='', part_kind='text'),
            ToolCallPart(tool_name='final_result', args='{"process_mode":"system_images"}', tool_call_id='call_cgkV1tPsu5ocH4comITuVV1A', part_kind='tool-call')
        ],
        usage=Usage(
            requests=1,
            request_tokens=2889,
            response_tokens=17,
            total_tokens=2906,
            details={'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0, 'cached_tokens': 0}
        ),
        model_name='gpt-4.1-mini-2025-04-14',
        timestamp=datetime.datetime(2025, 8, 12, 4, 33, 22, tzinfo=datetime.timezone.utc),
        kind='response',
    ),
    ModelRequest(
        parts=[
            ToolReturnPart(
                tool_name='final_result',
                content='Final result processed.',
                tool_call_id='call_cgkV1tPsu5ocH4comITuVV1A',
                timestamp=datetime.datetime(2025, 8, 12, 4, 33, 22, 502585, tzinfo=datetime.timezone.utc),
                part_kind='tool-return',
            )
        ],
        instructions=None,
        kind='request',
    )
  ]

[DouweM]

@sugawarayss I wonder if the (binary) image token usage is showing up on a field in the response that we don't currently translate to Usage.details. Can you please instrument HTTPX as well per https://ai.pydantic.dev/logfire/#monitoring-http-requests, and then see if you can find the usage field on the API response? If that doesn't include anything to account for the image tokens, this would be a bug on the Azure OpenAI side, as Pydantic AI does exactly the same thing for gpt-4o-mini and gpt-4.1-mini.

[sugawarayss]

@DouweM
i tried to Logfire, and it seems doesn't include image tokens in httpx response uage...

i sent same image files to gpt-4o-mini and gpt-4.1-mini (these are deployed on AzureOpenAI)

gpt-4o-mini usage (OK)

"usage": {
  "completion_tokens": 213,
  "completion_tokens_details": {
    "accepted_prediction_tokens": 0,
    "audio_tokens": 0,
    "reasoning_tokens": 0,
    "rejected_prediction_tokens": 0
  },
  "prompt_tokens": 26633,
  "prompt_tokens_details": {
    "audio_tokens": 0,
    "cached_tokens": 0
  },
  "total_tokens": 26846
}

gpt-4.1-mini usage (NG)

"usage": {
  "completion_tokens": 68,
  "completion_tokens_details": {
    "accepted_prediction_tokens": 0,
    "audio_tokens": 0,
    "reasoning_tokens": 0,
    "rejected_prediction_tokens": 0
  },
  "prompt_tokens": 1346,
  "prompt_tokens_details": {
    "audio_tokens": 0,
    "cached_tokens": 1024
  },
  "total_tokens": 1414
}

[DouweM]

@sugawarayss OK, then I suggest reaching out to your Azure rep about this, because it looks like a bug on their side rather than ours. You can extract the HTTP request body from Logfire and share it with them -- that'll likely be more useful than the Pydantic AI code used to generate the requests.

[DouweM]

I'll close this issue as this doesn't look like something we can fix, but if we learn something new I can reopen.