Cloudflare AI Gateway Fails to Parse Anthropic API usage Object for Token Counts and Costs

[binhpageai]

Proposed changes

Description

Cloudflare AI Gateway is not correctly parsing token usage data (input_tokens and output_tokens) from Anthropic API responses, resulting in inaccurate or missing token counts and cost estimates in the dashboard logs. Despite Anthropic’s API returning a usage object with these fields, the Gateway logs do not reflect the values, which impacts observability and cost tracking.

Steps to Reproduce

Set up a Cloudflare AI Gateway instance with an Anthropic endpoint (e.g., https://gateway.ai.cloudflare.com/v1/<account_id>/<gateway_id>/anthropic).
Send a POST request to the Anthropic API through the Gateway with a valid Anthropic API token:

curl https://gateway.ai.cloudflare.com/v1/<account_id>/<gateway_id>/anthropic -X POST \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <anthropic_api_token>' \
--data '{"model": "claude-3-5-sonnet-20241022", "messages": [{"role": "user", "content": "Test prompt"}]}'

Receive a response from Anthropic containing a usage object, such as:

{
  "type": "message",
  "role": "assistant",
  "model": "claude-3-5-sonnet-20241022",
  "stop_reason": "end_turn",
  "stop_sequence": null,
  "usage": {
    "input_tokens": 4144,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 0,
    "output_tokens": 475
  }
}

Check the Cloudflare AI Gateway dashboard (AI > AI Gateway > Logs) for the request.

Expected Behavior

The Gateway logs should display:
Input Tokens: 4144
Output Tokens: 475
Cost: Estimated based on Anthropic’s pricing (e.g., ~$0.0196 for Claude 3.5 Sonnet at $3/million input tokens and $15/million output tokens).
These values should match the input_tokens and output_tokens from the usage object in the Anthropic response.

Actual Behavior

The Gateway logs show:
Input Tokens: missing
Output Tokens: missing
Cost: missing
The token counts and costs are not parsed or displayed, despite the usage object being present in the response.

Suggested Fix

Update the Gateway’s parsing logic to recognize Anthropic’s usage fields (input_tokens, output_tokens) and map them to the dashboard’s token and cost metrics.
Ensure compatibility with Anthropic’s caching fields (cache_creation_input_tokens, cache_read_input_tokens) or ignore them if not supported, focusing only on core token counts.

Subject Matter

https://developers.cloudflare.com/ai-gateway/providers/anthropic/#_top

Content Location

https://developers.cloudflare.com/ai-gateway/providers/anthropic/#_top

Additional information

No response

[kodster28]

Howdy @binhpageai, passed this along internally... but closing out this issue b/c we're only a documentation team here (don't have any control over the broader dashboard).

[nickfujita]

@binhpageai
Just for reference in case anyone else comes across this. Was able to get the tokens and pricing back from the gateway by using the option compatibility = strict (assuming you are using the @ai-sdk/<provider> module.

const openai = createOpenAI({
  apiKey: this.env.OPENAI_API_KEY,
  baseURL: `${this.env.GATEWAY_BASE_URL}/openai`,
  compatibility: 'strict',
});