feat(groq): add prompt caching token usage details #33708
Conversation
github-actions
bot
added
integration
CodSpeed Performance ReportMerging #33708 will not alter performanceComparing Summary
Footnotes
|
| output_tokens = groq_token_usage.get("completion_tokens") or 0 | ||
| total_tokens = groq_token_usage.get("total_tokens") or input_tokens + output_tokens | ||
| input_token_details: dict = { | ||
| "cache_read": (groq_token_usage.get("prompt_tokens_details") or {}).get( |
There was a problem hiding this comment.
I can't find anywhere in Groq's docs where prompt_token_details are returned?
There was a problem hiding this comment.
I think it should be input_tokens_details
There was a problem hiding this comment.
Thanks for checking, It’s actually documented here:
https://console.groq.com/docs/prompt-caching#tracking-cache-usage
There was a problem hiding this comment.
@MshariAlaeena I think that might be wrong? See Slack
3 participants
Description:
Adds support for prompt caching usage metadata in ChatGroq. The integration now captures cached token information from the Groq API response and includes it in the
input_token_detailsfield of theusage_metadata.Changes:
_create_usage_metadata()helper function to centralize usage metadata creation logiccached_tokensfromprompt_tokens_detailsin API responses and maps toinput_token_details.cache_read_convert_chunk_to_message_chunk) and non-streaming (_create_chat_result) code pathsThis enables users to monitor prompt caching effectiveness when using Groq models with prompt caching enabled.
Issue: N/A
Dependencies: None