Question

I ran into a tricky issue using Anthropic's web_search with Pydantic AI that took some debugging to figure out. I wanted to share my findings and the workaround I came up with, in case this may help others who hit the same issue. I'm also curious if anyone has found better approaches, or if this is something the library could handle internally.

The issue: When using the web search builtin_tools with Anthropic models, and setting an output_type, I was getting ModelHTTPError: status_code: 400, model_name: claude-sonnet-4-20250514, body: {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.1: 'web_search' tool use with id '<some-id>' was found without a corresponding 'web_search_tool_result' block'}} fairly often. Capturing the messages when this happened, they would look something like this:

[
    ModelRequest(
        parts=[
            UserPromptPart(content='...', timestamp=...)
        ], 
        instructions='...'),
    ModelResponse(
        parts=[
            # succession of BuiltinToolCallPart, BuiltinToolReturnPart and TextPart blocks
            # ... ,
            BuiltinToolCallPart(tool_name='web_search', args={'query': '...'}, tool_call_id='some-id', provider_name='anthropic')
        ], 
        usage=Usage(...)
    ),
    ModelRequest(
        parts=[
            RetryPromptPart(content='Plain text responses are not permitted, please include your response in a tool call', tool_call_id=..., timestamp=...)
        ]
    )
]

My understanding of what's happening is the following: The last part in that ModelResponse in the middle is a BuiltinToolCallPart, without an associated BuiltinToolReturnPart (a query without it's answer). The server seems to be returning a response before it's finished. This results in a ModelRetry, since I've set output_type and that last part isn't a final output tool part. But when the framework tries to send this followup request, it's malformed (there's a "'web_search' tool use ... without a corresponding 'web_search_tool_result'"), so the server returns 400 and we get the ModelHTTPError.

My workaround: Use history_processors to detect that this is happening and "rewrite" the message history before the retry.

def remove_hanging_search(messages: list[ModelMessage]) -> list[ModelMessage]:
    """Remove incomplete web searches and provide better retry guidance."""
    if (
        len(messages) >= 2
        and isinstance(last_message := messages[-1], ModelRequest)
        and len(last_message.parts) == 1
        and isinstance(retry_part := last_message.parts[0], RetryPromptPart)
        and isinstance(last_response := messages[-2], ModelResponse)
        and isinstance(last_part := last_response.parts[-1], BuiltinToolCallPart)
        and last_part.tool_name == "web_search"
    ):
        # Remove the incomplete web search call
        _ = last_response.parts.pop()

        ## (Optional): Replace the generic retry message with a better one
        # retry_part.content = "Your search was interrupted. Proceed "

    return messages

Variations of the issue: I've also seen UnexpectedModelBehavior: Received empty model response errors, which I think stem from the same underlying issue (the server responding early, but this time with no TextParts). I fixed those just by tweaking the instructions to ask the model to provide commentary between searches (to prevent it from doing too many back to back searches without any text in between).

Additional Context

Python 3.12.8
pydantic-ai-slim 0.7.1
anthropic 0.64.0