-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Add FileSearchTool with support for OpenAI and Google
#3396
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 14 commits
4376b96
6cec96f
4c3fe56
3c8decf
2343679
666a1bb
7365e20
2ee21c9
deef1ec
18b4b86
11654ed
1542f5c
7d683b7
d8ef07d
6acbd76
380e25c
c83f125
8eba82d
b3a8930
19f32f9
00ea1ed
c6ed56c
9b5bb54
c2765ac
8286cd7
3011e05
bc278e8
5f694c9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,6 +12,7 @@ Pydantic AI supports the following built-in tools: | |
| - **[`UrlContextTool`][pydantic_ai.builtin_tools.UrlContextTool]**: Enables agents to pull URL contents into their context | ||
| - **[`MemoryTool`][pydantic_ai.builtin_tools.MemoryTool]**: Enables agents to use memory | ||
| - **[`MCPServerTool`][pydantic_ai.builtin_tools.MCPServerTool]**: Enables agents to use remote MCP servers with communication handled by the model provider | ||
| - **[`FileSearchTool`][pydantic_ai.builtin_tools.FileSearchTool]**: Enables agents to search through uploaded files using vector search (RAG) | ||
|
|
||
| These tools are passed to the agent via the `builtin_tools` parameter and are executed by the model provider's infrastructure. | ||
|
|
||
|
|
@@ -566,6 +567,93 @@ _(This example is complete, it can be run "as is")_ | |
| | `description` | ✅ | ❌ | | ||
| | `headers` | ✅ | ❌ | | ||
|
|
||
| ## File Search Tool | ||
|
|
||
| The [`FileSearchTool`][pydantic_ai.builtin_tools.FileSearchTool] enables your agent to search through uploaded files using vector search, providing a fully managed Retrieval-Augmented Generation (RAG) system. This tool handles file storage, chunking, embedding generation, and context injection into prompts. | ||
|
|
||
| ### Provider Support | ||
|
|
||
| | Provider | Supported | Notes | | ||
| |----------|-----------|-------| | ||
| | OpenAI Responses | ✅ | Full feature support. Requires files to be uploaded to vector stores via the [OpenAI Files API](https://platform.openai.com/docs/api-reference/files). Vector stores must be created and file IDs added before using the tool. | | ||
| | Google (Gemini) | ✅ | Requires files to be uploaded via the [Gemini Files API](https://ai.google.dev/gemini-api/docs/files). Files are automatically deleted after 48 hours. Supports up to 2 GB per file and 20 GB per project. Using built-in tools and function tools (including [output tools](output.md#tool-output)) at the same time is not supported; to use structured output, use [`PromptedOutput`](output.md#prompted-output) instead. | | ||
| | Anthropic | ❌ | Not supported | | ||
| | Groq | ❌ | Not supported | | ||
| | OpenAI Chat Completions | ❌ | Not supported | | ||
| | Bedrock | ❌ | Not supported | | ||
| | Mistral | ❌ | Not supported | | ||
| | Cohere | ❌ | Not supported | | ||
| | HuggingFace | ❌ | Not supported | | ||
| | Outlines | ❌ | Not supported | | ||
|
|
||
| ### Usage | ||
|
|
||
| #### OpenAI Responses | ||
|
|
||
| With OpenAI, you need to first upload files to a vector store, then reference the vector store IDs when using the `FileSearchTool`: | ||
|
|
||
| ```py {title="file_search_openai.py" test="skip"} | ||
| from pydantic_ai import Agent, FileSearchTool | ||
|
|
||
| agent = Agent( | ||
| 'openai-responses:gpt-5', | ||
| builtin_tools=[FileSearchTool(vector_store_ids=['vs_abc123'])] # (1) | ||
| ) | ||
|
|
||
| result = agent.run_sync('What information is in my documents about pydantic?') | ||
| print(result.output) | ||
| #> Based on your documents, Pydantic is a data validation library for Python... | ||
| ``` | ||
|
|
||
| 1. Replace `vs_abc123` with your actual vector store ID from the OpenAI API. | ||
|
|
||
| #### Google (Gemini) | ||
|
|
||
| With Gemini, you need to first upload files via the Files API, then reference the file resource names: | ||
|
||
|
|
||
| ```py {title="file_search_google.py" test="skip"} | ||
| from pydantic_ai import Agent, FileSearchTool | ||
|
|
||
| agent = Agent( | ||
| 'google-gla:gemini-2.5-flash', | ||
| builtin_tools=[FileSearchTool(vector_store_ids=['files/abc123'])] # (1) | ||
| ) | ||
|
|
||
| result = agent.run_sync('Summarize the key points from my uploaded documents.') | ||
| print(result.output) | ||
| #> The documents discuss the following key points: ... | ||
| ``` | ||
|
|
||
| 1. Replace `files/abc123` with your actual file resource name from the Gemini Files API. | ||
|
|
||
| !!! note "Gemini File Search API Status" | ||
| The File Search Tool for Gemini was announced on November 6, 2025. The implementation may require adjustment as the official `google-genai` SDK is updated to fully support this feature. | ||
|
||
|
|
||
| ### Configuration | ||
|
||
|
|
||
| The `FileSearchTool` accepts a list of vector store IDs: | ||
|
|
||
| - **OpenAI**: Vector store IDs created via the [OpenAI Files API](https://platform.openai.com/docs/api-reference/files) | ||
| - **Google**: File resource names from the [Gemini Files API](https://ai.google.dev/gemini-api/docs/files) | ||
|
||
|
|
||
| ```py {title="file_search_configured.py" test="skip"} | ||
| from pydantic_ai import Agent, FileSearchTool | ||
|
|
||
| agent = Agent( | ||
| 'openai-responses:gpt-5', | ||
| builtin_tools=[ | ||
| FileSearchTool( | ||
| vector_store_ids=['vs_store1', 'vs_store2'] # (1) | ||
| ) | ||
| ] | ||
| ) | ||
|
|
||
| result = agent.run_sync('Find information across all my document collections.') | ||
| print(result.output) | ||
| ``` | ||
|
|
||
| 1. You can provide multiple vector store IDs to search across different collections. | ||
|
|
||
| ## API Reference | ||
|
|
||
| For complete API documentation, see the [API Reference](api/builtin_tools.md). | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,6 +17,7 @@ | |
| 'ImageGenerationTool', | ||
| 'MemoryTool', | ||
| 'MCPServerTool', | ||
| 'FileSearchTool', | ||
| ) | ||
|
|
||
| _BUILTIN_TOOL_TYPES: dict[str, type[AbstractBuiltinTool]] = {} | ||
|
|
@@ -334,6 +335,30 @@ def unique_id(self) -> str: | |
| return ':'.join([self.kind, self.id]) | ||
|
|
||
|
|
||
| @dataclass(kw_only=True) | ||
| class FileSearchTool(AbstractBuiltinTool): | ||
| """A builtin tool that allows your agent to search through uploaded files using vector search. | ||
|
|
||
| This tool provides a fully managed Retrieval-Augmented Generation (RAG) system that handles | ||
| file storage, chunking, embedding generation, and context injection into prompts. | ||
|
|
||
| Supported by: | ||
|
|
||
| * OpenAI Responses | ||
| * Google (Gemini) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not vertex AI? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @DouweM Logan Kilpatrick responded on Twitter that Gemini File Search API is not yet available on Vertex AI.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks @shun-liang for checking! Correct, it's not available on Vertex AI yet according to Logan's response.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @gorkachea I just asked our contact at Google and they say:
It'd be nice to eventually support I do want to make it more explicit that Vertex is not support though. So let's add a |
||
| """ | ||
|
|
||
| vector_store_ids: list[str] | ||
| """List of vector store IDs to search through. | ||
|
|
||
| For OpenAI, these are the IDs of vector stores created via the OpenAI API. | ||
| For Google, these are file resource names that have been uploaded and processed. | ||
| """ | ||
|
|
||
| kind: str = 'file_search' | ||
| """The kind of tool.""" | ||
|
|
||
|
|
||
| def _tool_discriminator(tool_data: dict[str, Any] | AbstractBuiltinTool) -> str: | ||
| if isinstance(tool_data, dict): | ||
| return tool_data.get('kind', AbstractBuiltinTool.kind) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,7 +13,7 @@ | |
| from .. import UnexpectedModelBehavior, _utils, usage | ||
| from .._output import OutputObjectDefinition | ||
| from .._run_context import RunContext | ||
| from ..builtin_tools import CodeExecutionTool, ImageGenerationTool, UrlContextTool, WebSearchTool | ||
| from ..builtin_tools import CodeExecutionTool, FileSearchTool, ImageGenerationTool, UrlContextTool, WebSearchTool | ||
| from ..exceptions import UserError | ||
| from ..messages import ( | ||
| BinaryContent, | ||
|
|
@@ -342,6 +342,13 @@ def _get_tools(self, model_request_parameters: ModelRequestParameters) -> list[T | |
| tools.append(ToolDict(url_context=UrlContextDict())) | ||
| elif isinstance(tool, CodeExecutionTool): | ||
| tools.append(ToolDict(code_execution=ToolCodeExecutionDict())) | ||
| elif isinstance(tool, FileSearchTool): # pragma: no cover | ||
|
||
| # File Search Tool for Gemini API - tested via initialization tests | ||
|
||
| # The file_search tool uses file resource names (vector_store_ids) to search through uploaded files | ||
| # Note: This requires files to be uploaded via the Files API first | ||
| # The structure below is based on the Gemini File Search Tool announcement (Nov 2025) | ||
| # and may require adjustment when the official google-genai SDK is updated | ||
| tools.append(ToolDict(file_search={'file_names': tool.vector_store_ids})) # type: ignore[reportGeneralTypeIssues] | ||
|
||
| elif isinstance(tool, ImageGenerationTool): # pragma: no branch | ||
| if not self.profile.supports_image_output: | ||
| raise UserError( | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,7 +18,7 @@ | |
| from .._run_context import RunContext | ||
| from .._thinking_part import split_content_into_text_and_thinking | ||
| from .._utils import guard_tool_call_id as _guard_tool_call_id, now_utc as _now_utc, number_to_datetime | ||
| from ..builtin_tools import CodeExecutionTool, ImageGenerationTool, MCPServerTool, WebSearchTool | ||
| from ..builtin_tools import CodeExecutionTool, FileSearchTool, ImageGenerationTool, MCPServerTool, WebSearchTool | ||
| from ..exceptions import UserError | ||
| from ..messages import ( | ||
| AudioUrl, | ||
|
|
@@ -1071,8 +1071,10 @@ def _process_response( # noqa: C901 | |
| # Pydantic AI doesn't yet support the `codex-mini-latest` LocalShell built-in tool | ||
| pass | ||
| elif isinstance(item, responses.ResponseFileSearchToolCall): # pragma: no cover | ||
| # Pydantic AI doesn't yet support the FileSearch built-in tool | ||
| pass | ||
| # File Search Tool handling - requires actual OpenAI API responses with file_search_call | ||
|
||
| call_part, return_part = _map_file_search_tool_call(item, self.system) | ||
| items.append(call_part) | ||
| items.append(return_part) | ||
| elif isinstance(item, responses.response_output_item.McpCall): | ||
| call_part, return_part = _map_mcp_call(item, self.system) | ||
| items.append(call_part) | ||
|
|
@@ -1267,6 +1269,12 @@ def _get_builtin_tools(self, model_request_parameters: ModelRequestParameters) - | |
| type='approximate', **tool.user_location | ||
| ) | ||
| tools.append(web_search_tool) | ||
| elif isinstance(tool, FileSearchTool): # pragma: no cover | ||
| # File Search Tool configuration - tested via initialization tests | ||
|
||
| file_search_tool = responses.FileSearchToolParam( | ||
| type='file_search', vector_store_ids=tool.vector_store_ids | ||
| ) | ||
| tools.append(file_search_tool) | ||
| elif isinstance(tool, CodeExecutionTool): | ||
| has_image_generating_tool = True | ||
| tools.append({'type': 'code_interpreter', 'container': {'type': 'auto'}}) | ||
|
|
@@ -1404,6 +1412,7 @@ async def _map_messages( # noqa: C901 | |
| message_item: responses.ResponseOutputMessageParam | None = None | ||
| reasoning_item: responses.ResponseReasoningItemParam | None = None | ||
| web_search_item: responses.ResponseFunctionWebSearchParam | None = None | ||
| file_search_item: responses.ResponseFileSearchToolCallParam | None = None | ||
| code_interpreter_item: responses.ResponseCodeInterpreterToolCallParam | None = None | ||
| for item in message.parts: | ||
| if isinstance(item, TextPart): | ||
|
|
@@ -1473,6 +1482,23 @@ async def _map_messages( # noqa: C901 | |
| type='web_search_call', | ||
| ) | ||
| openai_messages.append(web_search_item) | ||
| elif ( # pragma: no cover | ||
|
||
| # File Search Tool - requires actual file_search responses in message history | ||
| item.tool_name == FileSearchTool.kind | ||
| and item.tool_call_id | ||
| and (args := item.args_as_dict()) | ||
| ): | ||
| # The cast is necessary because of incomplete OpenAI SDK types for FileSearchToolCall | ||
|
||
| file_search_item = cast( | ||
| responses.ResponseFileSearchToolCallParam, | ||
| { | ||
| 'id': item.tool_call_id, | ||
| 'action': args, | ||
| 'status': 'completed', | ||
| 'type': 'file_search_call', | ||
| }, | ||
| ) | ||
| openai_messages.append(file_search_item) | ||
| elif item.tool_name == ImageGenerationTool.kind and item.tool_call_id: | ||
| # The cast is necessary because of https://github.com/openai/openai-python/issues/2648 | ||
| image_generation_item = cast( | ||
|
|
@@ -1532,6 +1558,15 @@ async def _map_messages( # noqa: C901 | |
| and (status := content.get('status')) | ||
| ): | ||
| web_search_item['status'] = status | ||
| elif ( # pragma: no cover | ||
|
||
| # File Search Tool status update - only called from API-dependent paths | ||
|
||
| item.tool_name == FileSearchTool.kind | ||
| and file_search_item is not None | ||
| and isinstance(item.content, dict) # pyright: ignore[reportUnknownMemberType] | ||
| and (content := cast(dict[str, Any], item.content)) # pyright: ignore[reportUnknownMemberType] | ||
| and (status := content.get('status')) | ||
| ): | ||
| file_search_item['status'] = status | ||
| elif item.tool_name == ImageGenerationTool.kind: | ||
| # Image generation result does not need to be sent back, just the `id` off of `BuiltinToolCallPart`. | ||
| pass | ||
|
|
@@ -1845,6 +1880,12 @@ async def _get_event_iterator(self) -> AsyncIterator[ModelResponseStreamEvent]: | |
| yield self._parts_manager.handle_part( | ||
| vendor_part_id=f'{chunk.item.id}-call', part=replace(call_part, args=None) | ||
| ) | ||
| elif isinstance(chunk.item, responses.ResponseFileSearchToolCall): # pragma: no cover | ||
|
||
| # File Search Tool streaming - requires actual OpenAI streaming responses | ||
| call_part, _ = _map_file_search_tool_call(chunk.item, self.provider_name) | ||
| yield self._parts_manager.handle_part( | ||
| vendor_part_id=f'{chunk.item.id}-call', part=replace(call_part, args=None) | ||
| ) | ||
| elif isinstance(chunk.item, responses.ResponseCodeInterpreterToolCall): | ||
| call_part, _, _ = _map_code_interpreter_tool_call(chunk.item, self.provider_name) | ||
|
|
||
|
|
@@ -1913,6 +1954,18 @@ async def _get_event_iterator(self) -> AsyncIterator[ModelResponseStreamEvent]: | |
| elif isinstance(chunk.item, responses.ResponseFunctionWebSearch): | ||
| call_part, return_part = _map_web_search_tool_call(chunk.item, self.provider_name) | ||
|
|
||
| maybe_event = self._parts_manager.handle_tool_call_delta( | ||
| vendor_part_id=f'{chunk.item.id}-call', | ||
| args=call_part.args, | ||
| ) | ||
| if maybe_event is not None: # pragma: no branch | ||
| yield maybe_event | ||
|
|
||
| yield self._parts_manager.handle_part(vendor_part_id=f'{chunk.item.id}-return', part=return_part) | ||
| elif isinstance(chunk.item, responses.ResponseFileSearchToolCall): # pragma: no cover | ||
| # File Search Tool streaming response handling - requires actual OpenAI streaming responses | ||
| call_part, return_part = _map_file_search_tool_call(chunk.item, self.provider_name) | ||
|
|
||
| maybe_event = self._parts_manager.handle_tool_call_delta( | ||
| vendor_part_id=f'{chunk.item.id}-call', | ||
| args=call_part.args, | ||
|
|
@@ -2216,6 +2269,37 @@ def _map_web_search_tool_call( | |
| ) | ||
|
|
||
|
|
||
| def _map_file_search_tool_call( # pragma: no cover | ||
| # File Search Tool mapping - only called from API-dependent response processing paths | ||
|
||
| item: responses.ResponseFileSearchToolCall, | ||
| provider_name: str, | ||
| ) -> tuple[BuiltinToolCallPart, BuiltinToolReturnPart]: | ||
| args: dict[str, Any] | None = None | ||
|
|
||
| result = { | ||
| 'status': item.status, | ||
| } | ||
|
|
||
| # The OpenAI SDK has incomplete types for FileSearchToolCall.action | ||
|
||
| if action := item.action: # type: ignore[reportAttributeAccessIssue] | ||
| args = action.model_dump(mode='json') # type: ignore[reportUnknownMemberType] | ||
|
|
||
| return ( | ||
| BuiltinToolCallPart( | ||
| tool_name=FileSearchTool.kind, | ||
| tool_call_id=item.id, | ||
| args=args, # type: ignore[reportUnknownArgumentType] | ||
| provider_name=provider_name, | ||
| ), | ||
| BuiltinToolReturnPart( | ||
| tool_name=FileSearchTool.kind, | ||
| tool_call_id=item.id, | ||
| content=result, | ||
| provider_name=provider_name, | ||
| ), | ||
| ) | ||
|
|
||
|
|
||
| def _map_image_generation_tool_call( | ||
| item: responses.response_output_item.ImageGenerationCall, provider_name: str | ||
| ) -> tuple[BuiltinToolCallPart, BuiltinToolReturnPart, FilePart | None]: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,6 +20,7 @@ | |
| BuiltinToolReturnPart, | ||
| DocumentUrl, | ||
| FilePart, | ||
| FileSearchTool, | ||
| FinalResultEvent, | ||
| FunctionToolCallEvent, | ||
| FunctionToolResultEvent, | ||
|
|
@@ -3120,3 +3121,15 @@ def _generate_response_with_texts(response_id: str, texts: list[str]) -> Generat | |
| ], | ||
| } | ||
| ) | ||
|
|
||
|
|
||
| async def test_google_model_file_search_tool(allow_model_requests: None, google_provider: GoogleProvider): | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please have a look at the existing tests for built-in tools. We need to test:
We don't need the The same goes for OpenAI. If you can write the tests to have the same setup as other built-in tool tests (i.e. what I described above), I can run them, record the cassettes, and verify the messages look the way they should. |
||
| """Test that FileSearchTool can be configured with Google models.""" | ||
| m = GoogleModel('gemini-2.5-pro', provider=google_provider) | ||
| agent = Agent( | ||
| m, | ||
| builtin_tools=[FileSearchTool(vector_store_ids=['files/test123'])], | ||
| ) | ||
|
|
||
| # Just verify the agent initializes properly | ||
| assert agent is not None | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's link to the OpenAI docs here on how to do that, just to make sure they don't miss it in the table above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅ Done! Added links to the OpenAI and Gemini docs in both sections.