Anthropic prompt caching for tool response

[Wh1isper]

Description

In agentic scenarios, a single session (user query) typically involves dozens of tool invocations. We often add cache points to each tool response to ensure all tool responses are cached.

e.g.

[
user message
tool use 1
...
tool use 50
final result
]

We want tool use 50 to benefit from the caching of tool use 1 to tool use 49

References

• Add Anthropic prompt caching support #3363
• Anthropic prompt caching on Bedrock #3418

[Wh1isper]

For me, the simplest solution is to add a new option: something like anthropic_cache_all. If anthropic_cache_all=True, the last ModelRequest in each request would inject a cache point.

This is our current approach for previous version, and it doesn't even require introducing CachePoint into the code.

I'm not sure how many people need to control the cache for every Tool Response, but always caching is quite useful.

[mshavliuk]

Are there any plans to also add support for ttl param for Anthropic provider?

[Wh1isper]

Are there any plans to also add support for ttl param for Anthropic provider?

Thanks, I opened #3450 to add ttl. If #3450 is accepted, I would also support configuring ttl for cache all.

btw, manually configuring a 1-hour cache using CachePoint is more make sence? Curious about the scene of 1h cache

[AlirezaHaghi]

@DouweM I should say thank you too. But could you please support the same for Gemini? I really need that.
(I know this is not the right place to ask, but we have already had an issue to support prompt caching generally, which already seems ignored.)

[DouweM]

@Wh1isper Looking at this again, would it make sense to support CachePoint in ModelResponse.parts, so you can add it in a history processor? Or would the new flag work better for you?

[DouweM]

@AlirezaHaghi We currently support https://ai.pydantic.dev/api/models/google/#pydantic_ai.models.google.GoogleModelSettings.google_cached_content, is that not sufficient? If not, please create a new issue.

[Wh1isper]

would it make sense to support CachePoint in ModelResponse.parts

@DouweM I don't quite need the cache point in ModelResponse. My checkpoint is built based on ModelRequest.

[Wh1isper]

CachePoint in ModelResponse.parts

There appears to be only one reasonable scenario for placing a cache point in ModelResponse: when pre-filling the Context but not wanting to cache the message afterward.

We can discuss or implement this in a new issue/pull request. One counterintuitive thing here is that if a developer wants to add more CachePoints, our post-processing that preserves the last 4 CachePoints might silently delete the prefill CachePoint. Developers would then have to manually handle this using the history_processor.

This may brings us back to #3453. To be honest, I'm not quite sure.

[AlirezaHaghi]

@DouweM
It's hard for me to understand how to map the content in the docs with a config in the agent's creation-time.
When I read the Google doc, it seems we can cache a lot of things (including prompt, output JSON schema definition, tool definition, and so on). The point is that we should explicitly cache them (maybe I am wrong). But I'm not sure how to define these with Pydantic-ai model settings.

Let me describe to you the real challenge
My real use case is this: about 10M of text should be analysed and return a complex Pydantic model. in each API call, we spend more than 4k input tokens (because of the long prompt and complex output_type, pydantic model json schema)
So I would like to cache the prompt manually and the output model.

In the next step, I would like to leverage batch processing and combine it with a cache.

It is completely unclear to me if I can do that right now. Can I do all these only with google_cached_content

[DouweM]

There appears to be only one reasonable scenario for placing a cache point in ModelResponse: when pre-filling the Context but not wanting to cache the message afterward.

@Wh1isper OK, let's not do anything there then for now, as we don't currently support prefill anyway: #836

[DouweM]

@AlirezaHaghi Hmm, looking at https://ai.google.dev/gemini-api/docs/caching?lang=python#generate-content it looks like you'd need to explicitly create a cache with the instructions using the google genai SDK, outside of Pydantic AI, and then reference that cache using that model setting.

But you can't just tell Pydantic AI "cache the instructions/output type/some file" and have that happen automatically. We could potentially add support for that through model settings like the ones we've added for Anthropic. Feel free to file an issue -- we're derailing this one a bit :) Also note that newer models have implicit caching now: https://ai.google.dev/gemini-api/docs/caching