Skip to content

Conversation

@vrushankportkey
Copy link
Collaborator

Summary

Fixes an issue where prompt_tokens was missing from streaming responses for Anthropic models (both direct Anthropic API and via Vertex AI), causing billing/usage tracking to be incorrect.

Root cause: When streaming, the message_start event may not always include input_tokens. The code relied solely on this event to set prompt_tokens, resulting in undefined values that were omitted from the JSON response.

The fix:

  • Checks for input_tokens in the message_delta event as a fallback (per Anthropic docs, these are cumulative counts)
  • Defaults to 0 if neither source provides the value
  • Explicitly sets prompt_tokens in the usage object to ensure it's always present

Changes

  • src/providers/google-vertex-ai/chatComplete.ts - Vertex AI Claude streaming
  • src/providers/anthropic/chatComplete.ts - Direct Anthropic streaming

Test plan

  • Test streaming responses from Anthropic /v1/chat/completions - verify prompt_tokens is present
  • Test streaming responses from Vertex AI Claude /v1/chat/completions - verify prompt_tokens is present
  • Test with tool use / structured output calls specifically (the reported issue scenario)
  • Verify total_tokens calculation is still correct

Before (problematic response)

"usage": {
  "completion_tokens": 52,
  "prompt_tokens_details": {
    "cached_tokens": 0
  }
}

After (fixed response)

"usage": {
  "prompt_tokens": <value>,
  "completion_tokens": 52,
  "total_tokens": <calculated>,
  "prompt_tokens_details": {
    "cached_tokens": 0
  }
}

…sponses

When streaming responses from Anthropic models (direct or via Vertex AI),
the message_start event may not always include input_tokens. This caused
prompt_tokens to be undefined in the final response, breaking usage tracking
and billing calculations.

This fix:
- Checks for input_tokens in message_delta as a fallback
- Defaults to 0 if neither source provides the value
- Explicitly sets prompt_tokens in the usage object

Affects: /v1/chat/completions streaming for Anthropic and Vertex AI Claude models
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants