Include `cache_read_tokens` in OTel span attributes

[pedrovanrooij]

Question

I'm looking for confirmation that Pydantic AI allows for the default behavior from OpenAI regarding Prompt Caching.

I have run some tests using the default APIs from OpenAI and I'm able to see cached tokens being used which reduces the cost for my application. Perfoming the same exact action using the OpenAIResponsesModel does not use the cached prompts even if I try to add the prompt_cache_key as a extra_body parameter.

I reviewed the documentation and also checked current pull requests and issues and couldn't find any mentions to it this being supported.

Could you help me understand if Pydantic AI supports Prompt Caching from OpenAI?

Additional Context

No response

[DouweM]

Performing the same exact action using the OpenAIResponsesModel does not use the cached prompts even if I try to add the prompt_cache_key as a extra_body parameter.

Have you compared the HTTP requests generated by OpenAIResponsesModel and the OpenAI SDK? You can capture them using https://ai.pydantic.dev/logfire/#monitoring-http-requests. It'd be helpful to verify that the prompt_cache_key makes it in correctly, and to compare the prefixes of different requests to ensure they actually match to be cached.

[pedrovanrooij]

Hi @DouweM thanks for your reply and recommendation.

I went deeper into this to understand what is going on, while I don't know the full answer I might have a hint. I didn't use the prompt_cache_key as I wanted to test the default behavior for OpenAI both using OpenAIResponsesModel and the default Agent from the OpenAI Agents SDK.

I think it all hinders on how the information from OpenAI is parsed and shown in Logfire (or maybe the Pydantic AI SDK), I run both tests and printed in the console the output of the Usage, both have shown that cached_tokens were used but Logfire didn't show that.

I'm using Pydantic AI SDK version 1.0.18 and Agents SDK 0.3.3

Run with the OpenAI SDK

Logfire output

Usage object

Usage(requests=1, input_tokens=1124, input_tokens_details=InputTokensDetails(cached_tokens=1024), output_tokens=6147, output_tokens_details=OutputTokensDetails(reasoning_tokens=320), total_tokens=7271)

Run with the Pydantic AI SDK

Logfire output

Usage object

RunUsage(input_tokens=1124, cache_read_tokens=1024, output_tokens=6919, details={'reasoning_tokens': 320}, requests=1)

Question

Would it be something tied to PydanticAI or Logfire? And, is this something you could help figure out?

[DouweM]

@pedrovanrooij OK so the cache_read_tokens in the Pydantic AI RunUsage confirms that caching actually is working. Where are you saying it's not shown in Logfire?

[pedrovanrooij]

Hi @DouweM, let me try to explain. If you look at both screenshots taken from Logfire, there are no mentions to cache_read_tokens nor reasoning_tokens. I also looked at the raw information for that span and I don't see any references to those fields.

This makes me think that the total count of cached tokens is either not being captured by PydanticAI or is not being sent to Logfire.

So, I believe, it's not a matter of the framework having support for Prompt Caching anymore, I now think it's tied to reporting it so it's displayed appropriately in Logfire. This does change the scope of the question but wanted to ask if it's something I should follow up on with either you or the Logfire team.

Thank you for your help.

[DouweM]

@pedrovanrooij The reasoning_tokens in details are showing up in Logfire for me:

But you're right that the cache_read_tokens are not, due to this code:

pydantic-ai/pydantic_ai_slim/pydantic_ai/usage.py

Lines 68 to 82 in c5b1495

	def opentelemetry_attributes(self) -> dict[str, int]:
	"""Get the token usage values as OpenTelemetry attributes."""
	result: dict[str, int] = {}
	if self.input_tokens:
	result['gen_ai.usage.input_tokens'] = self.input_tokens
	if self.output_tokens:
	result['gen_ai.usage.output_tokens'] = self.output_tokens
	details = self.details
	if details:
	prefix = 'gen_ai.usage.details.'
	for key, value in details.items():
	# Skipping check for value since spec implies all detail values are relevant
	if value:
	result[prefix + key] = value
	return result

@alexmojaki Any concerns with including the other non-details Usage fields under gen_ai.usage.details.* as well?

[alexmojaki]

@pedrovanrooij your screenshots are coming from logfire.instrument_openai, not instrument_pydantic_ai.

@DouweM no that's fine, but some providers also include those fields under details already anyway.

[pedrovanrooij]

Thank you @DouweM and @alexmojaki for your help! I'm glad you spotted the issue with the cache_read_tokens!

I also noticed that I was ignoring the instrument_pydantic_ai as I had both that and the instrument_openai simultaneously. I fixed that in my code.

Look forward to updating Pydantic AI later on with the fix for cache_read_tokens!