Nudge Users to Select a Single OpenRouter Provider for Anthropic Models with Prompt Caching to Avoid Excessive Costs from Routing Policy

[samlam369]

I've greatly enjoyed using Roo Code via OpenRouter, especially with models like anthropic/claude-3.7-sonnet:thinking that support prompt caching.

On the OpenRouter > Activity page, I've noticed prompt caching can reduce the cost of requests by up to ~85%. This is a significant benefit for cost savings during iterative, back-and-forth interactions within a single Roo Code task.

However, OpenRouter's current routing policy doesn't distinguish between requests that are part of the same ongoing task. As a result, it may switch providers mid-task. When this happens, the prompt cache is lost for that provider, and the next request incurs the full, uncached cost again. On top of that, you pay a bit more for the first cache by an extra 25%.

For example, in a recent session of mine with 27 requests over 50 minutes, OpenRouter switched providers 4 times. This increased my total cost by 65% compared to sticking with a single provider:

Napkin math:

• The cost of any first request to a provider is 1.25x, and let's take 0.2x as the cost for subsequent requests (as a 80% discount benefited by caching).
• With one provider: 1 x 1.25 + 26 x 0.2 = 6.45
• With five providers (due to four switches): 5 x 1.25 + 22 x 0.2 = 10.65 (65% more expensive)

While dynamic routing may maximize uptime and reliability, it can unintentionally lead to much higher costs for users who rely on prompt caching.

This isn't a fault of Roo Code or OpenRouter, but the current default—letting OpenRouter dynamically select providers—can result in hidden costs that many users may not anticipate. Unless users are aware of this tradeoff, they might unknowingly pay significantly more than necessary.

Suggestion:
Roo Code could improve the user experience by clearly informing users about the cost implications of dynamic provider routing when using models that support prompt caching. A simple, visible reminder or prompt to select a specific provider for such models would help users make informed decisions and avoid unnecessary expenses.

Thank you for considering this suggestion!

Image showing how OpenRouter switches provider frequently:

Image showing how this particular model charges extra (by ~25%) every time switching to a new provider: