bug: fix inconsistent prompt_tokens normalization across Anthropic providers#1566
Open
BillionClaw wants to merge 1 commit intoPortkey-AI:mainfrom
Open
Conversation
…oviders The Bedrock provider correctly included cache tokens in prompt_tokens, but the direct Anthropic and Vertex AI providers did not. This caused inconsistent behavior when using prompt caching across different Anthropic provider implementations. Changes: - Anthropic provider (non-streaming): prompt_tokens now includes cache_creation_input_tokens + cache_read_input_tokens - Anthropic provider (streaming): message_start event now sets prompt_tokens to include cache tokens - Vertex AI provider (non-streaming): prompt_tokens now includes cache_creation_input_tokens + cache_read_input_tokens - Vertex AI provider (streaming): message_start event now sets prompt_tokens to include cache tokens Fixes Portkey-AI#1564
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Fixes #1564
The
prompt_tokenscount was inconsistent across Anthropic providers:prompt_tokensprompt_tokensprompt_tokensRoot Cause
The Bedrock provider correctly added
cache_creation_input_tokensandcache_read_input_tokenstoprompt_tokens, but the direct Anthropic and Vertex AI providers only usedinput_tokens.Fix
Updated both the Anthropic and Vertex AI providers to include cache tokens in
prompt_tokens:Changes Made:
prompt_tokenscalculation in response transformprompt_tokensin message_start eventprompt_tokenscalculation in response transformprompt_tokensin message_start eventVerification
The fix ensures consistent token counting behavior across all Anthropic provider implementations, matching the existing (correct) behavior in the Bedrock provider.
I have read and understood the CLA and agree to it.