Support for OpenAI Background Mode

[mattbrandman]

Description

OpenAI has a background mode seen here that is especially important when doing deep research both due to how it updates and its long running nature making keeping an HTTP connection open for the duration unlikely. Supporting background mode in our backend means having to build out a separate non-instrumented (or manually instrumented) set of functions that interact directly with OpenAI removing some of the usefulness of PydanticAI being the integration point for everything else.

The nature of background mode does not really have any equivalence with other providers which I understand makes doing it at request time in modelsettings slightly complicated. Maybe a new part could be returned as part of a modelresponse indicating this is an async request and then for each model (only openai to start it could handle what to do with that part on response). Alternatively that part could be given to a new function that returns the current state of that message, I realize this would be more interruptive as all models would need to implement.

References

No response

[DouweM]

The easiest way to implement this would be to add an OpenAIModelSettings.openai_background settings, and if enabled, pass background=True and implement this status-check loop inside OpenAIChatModel.request (and request_stream): https://platform.openai.com/docs/guides/background#polling-background-responses. That way the agent will still be long-running (and the rest of Pydantic AI doesn't need to know about backgrounded requests), but you wouldn't have to worry about connection timeouts and other issues.

The agent itself can be backgrounded like this: #2111 (comment), and if you can't trust the Python process to stay alive, you could use durable execution: https://ai.pydantic.dev/durable_execution/overview/

@mattbrandman Would that be sufficient for your use case?

[mattbrandman]

yup I think that would be perfect! Yeah we do use the durable execution already so maybe have it aware would make sense too?

[mattbrandman]

Although maybe it would be enough to the request check if the latest message has an id and background enabled, not entirely sure on the best way to do that. Though feels fairly straight forward based on other times I've looked at the code

Maybe it could return something that could be tracked / passed into subsequent calls if we're working with a durable execution and that process closes. That way when it restarts it has the ID to pass in and pydantic just rekicks off the rest of it?

[DouweM]

@mattbrandman Hmm, right now the entire Model.request method is wrapped in a Temporal activity (or the DBOS, Prefect equivalent), so if the durable execution is interrupted the entire activity/request will be run again. So if we handle this backgrounding and status-check loop entirely inside the OpenAIResponsesModel.request method and activity, we won't have a way to communicate the backgrounded task ID up to the workflow in order for it to be used when the workflow is interrupted and resumed.

That suggests that the request method (and activity) should immediately return something, e.g. a ModelResponse with a new incomplete=True field and the provider_response_id, and that the agent graph (ModelRequestNode) should then be responsible for calling request again with the incomplete ModelResponse passed in, in a loop until a complete response is received. We've had a need for something similar in #3051.

[mattbrandman]

that makes total sense!