Stream built-in tool calls from OpenAI, Google, Anthropic and return them on next request (required for OpenAI reasoning) (#2877)

Author: DouweM

docs/builtin-tools.md

@@ -26,10 +26,10 @@ making it ideal for queries that require up-to-date data.
 
 | Provider | Supported | Notes |
 |----------|-----------|-------|
-| OpenAI Responses | ✅ | Full feature support |
+| OpenAI Responses | ✅ | Full feature support. To include search results on the [`BuiltinToolReturnPart`][pydantic_ai.messages.BuiltinToolReturnPart], set the `openai_include_web_search_sources` setting to `True` on [`OpenAIResponsesModelSettings`][pydantic_ai.models.openai.OpenAIResponsesModelSettings]. |
 | Anthropic | ✅ | Full feature support |
+| Google | ✅ | No parameter support. No [`BuiltinToolCallPart`][pydantic_ai.messages.BuiltinToolCallPart] or [`BuiltinToolReturnPart`][pydantic_ai.messages.BuiltinToolReturnPart] is generated when streaming. Using built-in tools and user tools (including [output tools](output.md#tool-output)) at the same time is not supported; to use structured output, use [`PromptedOutput`](output.md#prompted-output) instead. |
 | Groq | ✅ | Limited parameter support. To use web search capabilities with Groq, you need to use the [compound models](https://console.groq.com/docs/compound). |
-| Google | ✅ | No parameter support. Google does not support using built-in tools and user tools (including [output tools](output.md#tool-output)) at the same time. To use structured output, use [`PromptedOutput`](output.md#prompted-output) instead. |
 | OpenAI Chat Completions | ❌ | Not supported |
 | Bedrock | ❌ | Not supported |
 | Mistral | ❌ | Not supported |
@@ -111,9 +111,9 @@ in a secure environment, making it perfect for computational tasks, data analysi
 
 | Provider | Supported | Notes |
 |----------|-----------|-------|
-| OpenAI | ✅ | |
+| OpenAI | ✅ | To include outputs on the [`BuiltinToolReturnPart`][pydantic_ai.messages.BuiltinToolReturnPart], set the `openai_include_code_execution_outputs` setting to `True` on [`OpenAIResponsesModelSettings`][pydantic_ai.models.openai.OpenAIResponsesModelSettings]. |
 | Anthropic | ✅ | |
-| Google | ✅ | Google does not support using built-in tools and user tools (including [output tools](output.md#tool-output)) at the same time. To use structured output, use [`PromptedOutput`](output.md#prompted-output) instead. |
+| Google | ✅ | Using built-in tools and user tools (including [output tools](output.md#tool-output)) at the same time is not supported; to use structured output, use [`PromptedOutput`](output.md#prompted-output) instead. |
 | Groq | ❌ | |
 | Bedrock | ❌ | |
 | Mistral | ❌ | |
@@ -140,7 +140,7 @@ allowing it to pull up-to-date information from the web.
 
 | Provider | Supported | Notes |
 |----------|-----------|-------|
-| Google | ✅ | Google does not support using built-in tools and user tools (including [output tools](output.md#tool-output)) at the same time. To use structured output, use [`PromptedOutput`](output.md#prompted-output) instead. |
+| Google | ✅ | No [`BuiltinToolCallPart`][pydantic_ai.messages.BuiltinToolCallPart] or [`BuiltinToolReturnPart`][pydantic_ai.messages.BuiltinToolReturnPart] is currently generated; please submit an issue if you need this. Using built-in tools and user tools (including [output tools](output.md#tool-output)) at the same time is not supported; to use structured output, use [`PromptedOutput`](output.md#prompted-output) instead. |
 | OpenAI | ❌ | |
 | Anthropic | ❌ | |
 | Groq | ❌ | |

docs/models/openai.md

@@ -112,37 +112,46 @@ agent = Agent(model)
 ...
 ```
 
+You can learn more about the differences between the Responses API and Chat Completions API in the [OpenAI API docs](https://platform.openai.com/docs/guides/migrate-to-responses).
+
+### Built-in tools
+
 The Responses API has built-in tools that you can use instead of building your own:
 
 - [Web search](https://platform.openai.com/docs/guides/tools-web-search): allow models to search the web for the latest information before generating a response.
+- [Code interpreter](https://platform.openai.com/docs/guides/tools-code-interpreter): allow models to write and run Python code in a sandboxed environment before generating a response.
 - [File search](https://platform.openai.com/docs/guides/tools-file-search): allow models to search your files for relevant information before generating a response.
 - [Computer use](https://platform.openai.com/docs/guides/tools-computer-use): allow models to use a computer to perform tasks on your behalf.
+- [Image generation](https://platform.openai.com/docs/guides/tools-image-generation): allow models to generate images based on a text prompt.
 
-You can use the `OpenAIResponsesModelSettings` class to make use of those built-in tools:
+Web search and Code interpreter are natively supported through the [Built-in tools](../builtin-tools.md) feature.
 
-```python
-from openai.types.responses import WebSearchToolParam  # (1)!
+Image generation is not currently supported. If you need this feature, please comment on [this issue](https://github.com/pydantic/pydantic-ai/issues/2140).
+
+File search and Computer use can be enabled by passing an [`openai.types.responses.FileSearchToolParam`](https://github.com/openai/openai-python/blob/main/src/openai/types/responses/file_search_tool_param.py) or [`openai.types.responses.ComputerToolParam`](https://github.com/openai/openai-python/blob/main/src/openai/types/responses/computer_tool_param.py) in the `openai_builtin_tools` setting on [`OpenAIResponsesModelSettings`][pydantic_ai.models.openai.OpenAIResponsesModelSettings]. They don't currently generate [`BuiltinToolCallPart`][pydantic_ai.messages.BuiltinToolCallPart] or [`BuiltinToolReturnPart`][pydantic_ai.messages.BuiltinToolReturnPart] parts in the message history, or streamed events; please submit an issue if you need native support for these built-in tools.
+
+```python{title="file_search_tool.py"}
+from openai.types.responses import FileSearchToolParam
 
 from pydantic_ai import Agent
 from pydantic_ai.models.openai import OpenAIResponsesModel, OpenAIResponsesModelSettings
 
 model_settings = OpenAIResponsesModelSettings(
-    openai_builtin_tools=[WebSearchToolParam(type='web_search_preview')],
+    openai_builtin_tools=[
+        FileSearchToolParam(
+            type='file_search',
+            vector_store_ids=['your-history-book-vector-store-id']
+        )
+    ],
 )
 model = OpenAIResponsesModel('gpt-4o')
 agent = Agent(model=model, model_settings=model_settings)
 
-result = agent.run_sync('What is the weather in Tokyo?')
+result = agent.run_sync('Who was Albert Einstein?')
 print(result.output)
-"""
-As of 7:48 AM on Wednesday, April 2, 2025, in Tokyo, Japan, the weather is cloudy with a temperature of 53°F (12°C).
-"""
+#> Albert Einstein was a German-born theoretical physicist.
 ```
 
-1. The file search tool and computer use tool can also be imported from `openai.types.responses`.
-
-You can learn more about the differences between the Responses API and Chat Completions API in the [OpenAI API docs](https://platform.openai.com/docs/guides/responses-vs-chat-completions).
-
 #### Referencing earlier responses
 
 The Responses API supports referencing earlier model responses in a new request using a `previous_response_id` parameter, to ensure the full [conversation state](https://platform.openai.com/docs/guides/conversation-state?api-mode=responses#passing-context-from-the-previous-response) including [reasoning items](https://platform.openai.com/docs/guides/reasoning#keeping-reasoning-items-in-context) are kept in context. This is available through the `openai_previous_response_id` field in

pydantic_ai_slim/pydantic_ai/_agent_graph.py

@@ -545,21 +545,22 @@ async def _run_stream(  # noqa: C901
             # Ensure that the stream is only run once
 
             async def _run_stream() -> AsyncIterator[_messages.HandleResponseEvent]:  # noqa: C901
-                texts: list[str] = []
+                text = ''
                 tool_calls: list[_messages.ToolCallPart] = []
                 thinking_parts: list[_messages.ThinkingPart] = []
 
                 for part in self.model_response.parts:
                     if isinstance(part, _messages.TextPart):
-                        # ignore empty content for text parts, see #437
-                        if part.content:
-                            texts.append(part.content)
+                        text += part.content
                     elif isinstance(part, _messages.ToolCallPart):
                         tool_calls.append(part)
                     elif isinstance(part, _messages.BuiltinToolCallPart):
-                        yield _messages.BuiltinToolCallEvent(part)
+                        # Text parts before a built-in tool call are essentially thoughts,
+                        # not part of the final result output, so we reset the accumulated text
+                        text = ''
+                        yield _messages.BuiltinToolCallEvent(part)  # pyright: ignore[reportDeprecated]
                     elif isinstance(part, _messages.BuiltinToolReturnPart):
-                        yield _messages.BuiltinToolResultEvent(part)
+                        yield _messages.BuiltinToolResultEvent(part)  # pyright: ignore[reportDeprecated]
                     elif isinstance(part, _messages.ThinkingPart):
                         thinking_parts.append(part)
                     else:
@@ -572,9 +573,9 @@ async def _run_stream() -> AsyncIterator[_messages.HandleResponseEvent]:  # noqa
                 if tool_calls:
                     async for event in self._handle_tool_calls(ctx, tool_calls):
                         yield event
-                elif texts:
+                elif text:
                     # No events are emitted during the handling of text responses, so we don't need to yield anything
-                    self._next_node = await self._handle_text_response(ctx, texts)
+                    self._next_node = await self._handle_text_response(ctx, text)
                 elif thinking_parts:
                     # handle thinking-only responses (responses that contain only ThinkingPart instances)
                     # this can happen with models that support thinking mode when they don't provide
@@ -593,9 +594,16 @@ async def _run_stream() -> AsyncIterator[_messages.HandleResponseEvent]:  # noqa
                     if isinstance(ctx.deps.output_schema, _output.TextOutputSchema):
                         for message in reversed(ctx.state.message_history):
                             if isinstance(message, _messages.ModelResponse):
-                                last_texts = [p.content for p in message.parts if isinstance(p, _messages.TextPart)]
-                                if last_texts:
-                                    self._next_node = await self._handle_text_response(ctx, last_texts)
+                                text = ''
+                                for part in message.parts:
+                                    if isinstance(part, _messages.TextPart):
+                                        text += part.content
+                                    elif isinstance(part, _messages.BuiltinToolCallPart):
+                                        # Text parts before a built-in tool call are essentially thoughts,
+                                        # not part of the final result output, so we reset the accumulated text
+                                        text = ''  # pragma: no cover
+                                if text:
+                                    self._next_node = await self._handle_text_response(ctx, text)
                                     return
 
                     raise exceptions.UnexpectedModelBehavior('Received empty model response')
@@ -655,11 +663,9 @@ def _handle_final_result(
     async def _handle_text_response(
         self,
         ctx: GraphRunContext[GraphAgentState, GraphAgentDeps[DepsT, NodeRunEndT]],
-        texts: list[str],
+        text: str,
     ) -> ModelRequestNode[DepsT, NodeRunEndT] | End[result.FinalResult[NodeRunEndT]]:
         output_schema = ctx.deps.output_schema
-
-        text = '\n\n'.join(texts)
         try:
             run_context = build_run_context(ctx)
             if isinstance(output_schema, _output.TextOutputSchema):

pydantic_ai_slim/pydantic_ai/_cli.py

@@ -356,7 +356,7 @@ def handle_slash_command(
         except IndexError:
             console.print('[dim]No output available to copy.[/dim]')
         else:
-            text_to_copy = '\n\n'.join(part.content for part in parts if isinstance(part, TextPart))
+            text_to_copy = ''.join(part.content for part in parts if isinstance(part, TextPart))
             text_to_copy = text_to_copy.strip()
             if text_to_copy:
                 pyperclip.copy(text_to_copy)