feat: add prompt caching support for LiteLLM (#5791)

[MuriloFP]

Related GitHub Issue

Closes: #5791

Roo Code Task Context (Optional)

No Roo Code task context for this PR

Description

This PR implements prompt caching support for LiteLLM, allowing users to benefit from reduced costs and improved response times when using models that support prompt caching (like Claude 3.7).

Key implementation details:

• Added litellmUsePromptCache boolean option to provider settings schema
• Modified the LiteLLM handler to apply cache control headers to system messages and the last two user messages when the feature is enabled
• Enhanced usage tracking to properly capture cache read/write tokens from LiteLLM responses, including alternative field names
• Added UI checkbox that only appears when the selected model supports prompt caching
• Reused existing translation keys (enablePromptCaching and enablePromptCachingTitle) to maintain consistency across all supported languages

Design choices:

• Followed the same caching pattern as the Anthropic provider (caching system prompt + last 2 user messages)
• Made the feature opt-in via a checkbox to give users control over when caching is used
• Kept the test focused on the critical functionality to follow project patterns

Test Procedure

Automated Testing:

• Added unit test in src/api/providers/__tests__/lite-llm.spec.ts that verifies:
  • Cache control headers are properly added when litellmUsePromptCache is enabled
  • Cache tokens are correctly tracked in usage data
  • The feature respects the model's prompt caching support capability

Manual Testing Steps:

1. Configure LiteLLM as your provider with a model that supports prompt caching (e.g., Claude 3.7)
2. Navigate to Settings > Providers > LiteLLM
3. Verify the "Enable prompt caching" checkbox appears
4. Enable the checkbox and save settings
5. Start a conversation and monitor the LiteLLM logs/dashboard
6. Verify that cache hits/misses are being recorded
7. Check that the usage tracking in Roo Code shows cache read/write tokens

Test Command:

cd src && npx vitest run api/providers/__tests__/lite-llm.spec.ts

Pre-Submission Checklist

• Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
• Scope: My changes are focused on the linked issue (one major feature/fix per PR).
• Self-Review: I have performed a thorough self-review of my code.
• Testing: New and/or updated tests have been added to cover my changes (if applicable).
• Documentation Impact: I have considered if my changes require documentation updates (see "Documentation Updates" section below).
• Contribution Guidelines: I have read and agree to the Contributor Guidelines.

Screenshots / Videos

Before: The LiteLLM settings page shows only Base URL, API Key, and Model selection.

After: When a model that supports prompt caching is selected, an additional "Enable prompt caching" checkbox appears with a description.

Note: The checkbox only appears for models that have supportsPromptCache: true in their model info.

Documentation Updates

• No documentation updates are required.

The feature is self-explanatory through the UI, using existing translation keys that are already documented.

Additional Notes

This implementation follows the same approach as the referenced Cline commit but adapts it to RooCode's architecture. The main difference is that we reuse existing translation keys instead of creating new ones, which ensures all languages are supported without additional translation work.

Get in Touch

@MuriloFP

---

Important

Adds prompt caching support for LiteLLM, including schema updates, handler modifications, UI changes, and tests.

• Behavior:
  • Adds litellmUsePromptCache boolean to provider settings schema in provider-settings.ts.
  • Modifies LiteLLMHandler in lite-llm.ts to add cache control headers to system and last two user messages if caching is enabled.
  • Tracks cache read/write tokens in LiteLLMHandler.
• UI:
  • Adds a checkbox for enabling prompt caching in LiteLLM.tsx, visible only for models supporting caching.
• Testing:
  • Adds unit test in lite-llm.spec.ts to verify cache control headers and token tracking when caching is enabled.

^{This description was created by for d460f43. You can customize this summary. It will automatically update as commits are pushed.}

[ellipsis-dev]

Consider adding an assertion for the second last user message as well, to fully verify that cache control headers are applied to both the last two user messages.

[MuriloFP]

Updated PR with improved prompt caching implementation

Based on feedback from issue #5791, I've updated the prompt caching implementation to properly handle multi-turn conversations.

Changes made:

1. Fixed prompt caching for subsequent messages: The previous implementation only cached the system prompt and first user message. Now it properly caches all messages in a conversation.

2. Improved cache control structure:

   • System messages now use a structured format with applied to content items
   • User messages handle both string and array content types correctly
   • Cache control is applied at the content level, not just the message level

3. TypeScript compatibility: Added type assertions to handle the property that's not in OpenAI's type definitions

Technical details:

The key issue was that LiteLLM needs cache control to be applied more granularly - specifically to content items within messages, similar to how Anthropic's native API handles it. The flat approach of spreading cache control at the message level doesn't work correctly for subsequent messages in a conversation.

This implementation now follows the pattern used in Cline's rather than their , which ensures proper caching throughout multi-turn conversations.

All tests are passing and the linter issues have been resolved.

[daniel-lxs]

LGTM

[daniel-lxs]

Hey @MuriloFP Can you confirm this follows the documentation on https://docs.litellm.ai/docs/proxy/caching#usage? I think I see some inconsistencies, liteLLM doesn't seem to follow the usual OpenAI-style caching but I might be wrong.

[steve-gore-snapdocs]

I believe https://docs.litellm.ai/docs/proxy/caching#usage refers to caching in LiteLLM when the proxy receives identical requests. In that case no request is sent on to the LLM Model and LiteLLM's cached response is returned. This PR adds caching directives to the messages that are proxied by LiteLLM to the LLM Model.

[daniel-lxs]

Thank you @MuriloFP and @steve-gore-snapdocs for the clarification!