Add basic support for UploadedFile UserContent #2611

tarruda · 2025-08-19T18:39:41Z

@DouweM here's some initial support for #2574

The UploadedFile user content is simply wrapping an opaque reference to some return value of provider-specific file upload API, which is then validated in the corresponding model _map_user_prompt.

I've only added support for Google and OpenAI, but I believe the API should be flexible enough to add support for other providers. I started working on Anthropic, but decided to leave it out for now as the official SDK doesn't support this feature yet, and I was having trouble referencing it using the SDK data objects.

I've opted to not implement a Provider.upload_file abstraction, as the options can be different across providers and I would need to get more familiar with pydantic-ai before feeling confident enough to design a proper abstraction (Can follow up with another PR later!)

One caveat with the tests: The VCR framework apparently doesn't support requests containing binary content, so I had to turn off for uploading files. This is how I proceeded to add the tests:

Wrote code to upload the file (a new smiley pdf which I've added to tests/assets/smiley.pdf) and turned vcr off.
Ran the test with a print statement to show the return value
Commented the code to upload the file (left it as reference for later)
Re-ran the tests with recording on, and with a literal provider-specific file object with the same id I had previously uploaded.

Since this is just a recording and we are only verifying that we can reference an uploaded file, it probably doesn't matter much that we are not actually running the upload request for now. This can be changed later when VCR is fixed to support this type of request.

Close #2574

pydantic_ai_slim/pydantic_ai/models/google.py

This wraps an opaque reference to a provider-specific representation of an uploaded file.

tarruda · 2025-08-19T19:26:01Z

@DouweM CI still failing on code coverage. I will fix it, but first I'd love some feedback on the API . LMK if you agree with the choices or if I should make some adjustments!

DouweM

@tarruda I just noticed I never submitted the review I did many weeks ago 🤦🏻

pydantic_ai_slim/pydantic_ai/messages.py

pydantic_ai_slim/pydantic_ai/models/google.py

pydantic_ai_slim/pydantic_ai/models/openai.py

tests/models/test_google.py

DouweM · 2025-09-01T22:25:55Z

tests/models/test_openai.py

+    provider = OpenAIProvider(api_key=openai_api_key)
+    m = OpenAIModel('gpt-4o', provider=provider)
+    # VCR recording breaks when dealing with openai file upload request due to
+    # binary contents. For that reason, we have manually run once the upload


Binary appears to be supported: https://github.com/kevin1024/vcrpy/blob/d50f3385a6828280def801ac7f544fe04a37e39c/tests/unit/test_json_serializer.py#L7

Can you share the error you were seeing?

Here's what I get when I uncomment the code to upload on the google test (with vcr enabled):

| AssertionError +---------------- 2 ---------------- | Traceback (most recent call last): | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2266, in run_test | self.get_loop().run_until_complete( | File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete | return future.result() | ^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2226, in _call_in_runner_task | return await future | ^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2193, in _run_tests_and_fixtures | retval = await coro | ^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/tests/models/test_google.py", line 2787, in test_uploaded_file_input | google_file = client.files.upload( | ^^^^^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/google/genai/files.py", line 484, in upload | return_file = self._api_client.upload_file( | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/google/genai/_api_client.py", line 1438, in upload_file | return self._upload_fd( | ^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/google/genai/_api_client.py", line 1500, in _upload_fd | response = self._httpx_client.request( | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/httpx/_client.py", line 825, in request | return self.send(request, auth=auth, follow_redirects=follow_redirects) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/httpx/_client.py", line 914, in send | response = self._send_handling_auth( | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth | response = self._send_handling_redirects( | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects | response = self._send_single_request(request) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/vcr/stubs/httpx_stubs.py", line 200, in _inner_send | return _sync_vcr_send(cassette, real_send, *args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/vcr/stubs/httpx_stubs.py", line 186, in _sync_vcr_send | vcr_request, response = _shared_vcr_send(cassette, real_send, *args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/vcr/stubs/httpx_stubs.py", line 117, in _shared_vcr_send | vcr_request = _make_vcr_request(real_request, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/thiago/code/pydantic-ai/.venv/lib/python3.12/site-packages/vcr/stubs/httpx_stubs.py", line 108, in _make_vcr_request | body = httpx_request.read().decode("utf-8") | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 72: invalid start byte

pydantic_ai_slim/pydantic_ai/messages.py

github-actions · 2025-10-30T14:00:51Z

This PR is stale, and will be closed in 3 days if no reply is received.

github-actions · 2025-11-03T14:00:54Z

Closing this PR as it has been inactive for 10 days.

tarruda · 2025-11-25T13:17:11Z

@tarruda I just noticed I never submitted the review I did many weeks ago 🤦🏻

@DouweM and I completely missed the notification of you submitting the review 🤦🏻

Can you re-open the PR? I can address the comments and rebase

DouweM · 2025-11-25T13:18:51Z

@tarruda Thank you, reopened!

- map UploadedFile to provider-friendly structures: Google uses file_data parts; OpenAI accepts file IDs or objects with ids - document provider expectations for UploadedFile in code and input docs - add tests and cassette adjustments to cover file ID/URI handling for OpenAI and Google

- add an `UploadedFilePart` schema and emit uploaded-file metadata in OTEL user prompt parts, including file references when allowed - derive stable identifiers for `UploadedFile` objects with optional overrides for clearer telemetry - silence the pyright private-usage warning in the Google uploaded file test

tarruda · 2025-11-26T17:18:44Z

@DouweM I merged main (lots of conflicts 😓)

Also addressed your review!

DouweM · 2025-11-26T23:02:27Z

docs/input.md

 #> The document discusses...
 ```

+## Uploaded files


I just merged #3492 which (among other things) added an Uploaded Files section as well :)

Can you merge main and update that example to use the UploadedFile object? Keeping the section above the "user-side ..." section makes sense to me.

DouweM · 2025-11-26T23:02:52Z

docs/input.md


+## Uploaded files
+
+Use [`UploadedFile`][pydantic_ai.UploadedFile] when you've already uploaded content to the model provider.


Related to the above, let's include examples of how to do that for all providers

DouweM · 2025-11-26T23:03:44Z

docs/input.md

+
+- [`OpenAIChatModel`][pydantic_ai.models.openai.OpenAIChatModel] and [`OpenAIResponsesModel`][pydantic_ai.models.openai.OpenAIResponsesModel] accept an `openai.types.FileObject` or a file ID string returned by the OpenAI Files API.
+- [`GoogleModel`][pydantic_ai.models.google.GoogleModel] accepts a `google.genai.types.File` or a file URI string from the Gemini Files API.
+- Other models currently raise `NotImplementedError` when they receive an `UploadedFile`.


Let's support Anthropic as well: https://platform.claude.com/docs/en/build-with-claude/files

Does anthropic provide a client-side SDK for this? In the link I only see it being done with http requests.

@tarruda It's not super discoverable, but all of those code samples have a "Shell" dropdown that also has a "Python" option. So yes there's an SDK for uploading files, and their objects for passing file URLs and binary data also have a file_id field that maps to the ID returned by the file upload SDK.

DouweM · 2025-11-26T23:03:50Z

docs/input.md

+- [`GoogleModel`][pydantic_ai.models.google.GoogleModel] accepts a `google.genai.types.File` or a file URI string from the Gemini Files API.
+- Other models currently raise `NotImplementedError` when they receive an `UploadedFile`.
+
+```py {title="uploaded_file_input.py" test="skip" lint="skip"}


Please don't skip linting

DouweM · 2025-11-26T23:04:02Z

docs/input.md

+result = agent.run_sync(
+    [
+        'Give me a short description of this image',
+        UploadedFile(file='file-abc123'),  # file-abc123 is a file ID returned by the provider


Let's update the example to be more "real"

Can you elaborate?

I just meant that we can actually show the code for uploading a file using the provider SDK, and then passing in the return object/ID here instead of a fake ID

DouweM · 2025-11-26T23:08:21Z

pydantic_ai_slim/pydantic_ai/models/google.py

+        if isinstance(file, File):
+            file_uri = file.uri
+            mime_type = file.mime_type
+            display_name = getattr(file, 'display_name', None)


Why getattr instead of a regular attr read?

DouweM · 2025-11-26T23:09:23Z

pydantic_ai_slim/pydantic_ai/models/openai.py

    )


+def _map_uploaded_file(uploaded_file: UploadedFile, _provider: Provider[Any]) -> str:


Doesn't look like we need the provider?

DouweM · 2025-11-26T23:10:39Z

pydantic_ai_slim/pydantic_ai/models/openai.py

+    if isinstance(file, FileObject):
+        return file.id
+
+    file_id = getattr(file, 'id', None)


I don't think we need to support arbitrary objects with an id; rather just the types allowed on the future OpenAIUploadedFile: str and FileObject

DouweM · 2025-11-26T23:11:08Z

tests/models/test_google.py

+async def test_uploaded_file_input(allow_model_requests: None, google_provider: GoogleProvider):
+    m = GoogleModel('gemini-2.5-flash', provider=google_provider)
+    # VCR recording breaks when dealing with openai file upload request due to
+    # binary contents. For that reason, we have manually run once the upload


Can you try if this has been fixed? I think we have some VCRs containing binary files already

Unless it has been fixed in the latest main merge today, I'm certain the bug is still present. I tried after fixing the conflicts.

This is very easy to reproduce locally:

Uncomment the block which uploads the file

Run:

uv run pytest tests/models/test_google.py -k uploaded_file_input --record-mode=all

DouweM · 2025-11-26T23:11:32Z

tests/test_uploaded_file.py

Please move to test_messages for consistency

github-actions · 2025-12-07T14:00:44Z

This PR is stale, and will be closed in 3 days if no reply is received.

tarruda commented Aug 19, 2025

View reviewed changes

pydantic_ai_slim/pydantic_ai/models/google.py Outdated Show resolved Hide resolved

tarruda force-pushed the support-file-uploads branch from d6f2bb3 to 4fd4a88 Compare August 19, 2025 18:46

tarruda added 2 commits August 19, 2025 15:47

Add UploadedFile UserContent

2cb4086

This wraps an opaque reference to a provider-specific representation of an uploaded file.

Implement support for UploadedFile in OpenAIModel

6abc260

tarruda force-pushed the support-file-uploads branch 3 times, most recently from 2917ac8 to 8a837b5 Compare August 19, 2025 19:11

tarruda added 4 commits August 19, 2025 16:17

Support UploadedFile for OpenAI models

2c3c6a0

Add test for OpenAI UploadedFile

e05ea0c

Support UploadedFile for google genai models

af6b2a1

Add test for Google UploadedFile

ffa6a57

tarruda force-pushed the support-file-uploads branch from 8a837b5 to 8140f52 Compare August 19, 2025 19:18

Add placeholder for handling UploadedFile in bedrock/gemini/huggingface

0d6e486

tarruda force-pushed the support-file-uploads branch from 8140f52 to 0d6e486 Compare August 19, 2025 19:22

DouweM self-assigned this Sep 1, 2025

DouweM added the awaiting author revision label Sep 1, 2025

DouweM requested changes Sep 30, 2025

View reviewed changes

github-actions bot added the Stale label Oct 30, 2025

github-actions bot closed this Nov 3, 2025

DouweM mentioned this pull request Nov 6, 2025

Add Support for OpenAI and Gemini File Search Tools #3358

Open

DouweM reopened this Nov 25, 2025

DouweM removed the Stale label Nov 25, 2025

DouweM mentioned this pull request Nov 25, 2025

Support referencing file_ids previously uploaded to model providers. #2574

Open

tarruda added 2 commits November 26, 2025 11:11

Merge branch 'main' into support-file-uploads

00fdb5a

tarruda added 2 commits November 26, 2025 12:35

Regenerate UploadedFile tests with dummy.pdf

6e8dd1d

tarruda requested a review from DouweM November 26, 2025 17:18

tarruda added 2 commits November 26, 2025 14:36

Merge branch 'main' into support-file-uploads

eb0d4f3

Increase coverage for uploading files

d55377f

DouweM requested changes Nov 26, 2025

View reviewed changes

DouweM mentioned this pull request Dec 2, 2025

Pass s3:// file URLs directly to API in BedrockConverseModel #3621

Open

github-actions bot added the Stale label Dec 7, 2025


		## Uploaded files

		Use [`UploadedFile`][pydantic_ai.UploadedFile] when you've already uploaded content to the model provider.

		)


		def _map_uploaded_file(uploaded_file: UploadedFile, _provider: Provider[Any]) -> str:

Add basic support for UploadedFile UserContent #2611

Are you sure you want to change the base?

Add basic support for UploadedFile UserContent #2611

Conversation

tarruda commented Aug 19, 2025

Uh oh!

Uh oh!

tarruda commented Aug 19, 2025

Uh oh!

DouweM left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Nov 3, 2025

Uh oh!

tarruda commented Nov 25, 2025

Uh oh!

DouweM commented Nov 25, 2025

Uh oh!

tarruda commented Nov 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants