🚀 Feature Proposal: Add Prompt Caching Compatibility for OpenAI-Compatible Providers

[andrewgmercader]

Summary

The current code works perfectly with OpenAI-compatible APIs (see example below).
It would be a simple and valuable improvement to add prompt caching compatibility for OpenAI-compatible providers, allowing for more efficient and cost-effective usage.

Note: The Kilo Code developers are the ones who best know where caching could be implemented in the codebase, so integrating this feature should be quite straightforward in principle.

Example Code (with mocked key and URL)

import openai

client = openai.OpenAI(
    api_key="test_key_1234567890",
    base_url="https://mocked.api.provider/v1"
)

response = client.chat.completions.create(
    model="vertex_ai/claude-sonnet-4-20250514",
    messages=[
        {
            "role": "system",
            "content": [
                {"type": "text", "text": "You are a helpful assistant."},
                {
                    "type": "text",
                    "text": "Some input text here.",
                    "cache_control": {"type": "ephemeral"}  # <- hack for caching
                }
            ]
        },
        {"role": "user", "content": "Summarize the above in 20 words."}
    ],
    extra_body={}  # trick: allows unknown fields to pass through
)

print(response.choices[0].message.content)

Motivation

• Efficiency: Prompt caching can reduce repeated computation and API costs.
• Compatibility: Many OpenAI-compatible providers (e.g., Anthropic, Vertex AI, etc.) could benefit from a standardized caching mechanism.
• Flexibility: Developers can control cache behavior per message or request.

Proposed Solution

• Add a cache_control parameter to the message schema, as shown above.
• Document how to use this parameter for OpenAI-compatible providers.
• Optionally, add a helper function or middleware to manage prompt caching transparently.
• Since the Kilo Code maintainers are most familiar with the codebase, identifying the optimal place to implement caching logic should be straightforward.

Implementation Notes

• The code already supports passing custom fields via extra_body={}.
• The cache_control field is ignored by the OpenAI SDK but can be used by compatible providers or custom gateways.
• This feature could be implemented as a lightweight wrapper or decorator.

Additional Context

• This approach is non-breaking and backward compatible.
• It follows open source best practices by being simple, well-documented, and easy to adopt.