feat: add caching if prompt request fails

[Matthieu-OD]

Add Prompt Caching Functionality

Changes

• Implemented a prompt caching system using a separate class SharedCachePrompt

Implementation Details

• Cache keys are generated in three formats:

• id

• name

• tuple(name, version)

• Caching logic implemented in both sync and async get_prompt methods

• On each successful get prompt we will cache the prompt with keys id, name, tuple(name, version)

• Added fallback to cached prompts when API calls fail, with warning logs

• To keep in mind this cache will be cleared each time the application Is restarted

Test Commands

Test Code

from fastapi import FastAPI, HTTPException
from typing import Optional
from literalai import LiteralClient

app = FastAPI()
client = LiteralClient(
    url="http://localhost:3000", api_key="my-initial-api-key"
)

@app.get("/prompt")
async def get_prompt(
    id: Optional[str] = None,
    name: Optional[str] = None,
    version: Optional[int] = None,
    should_fail: bool = False  # For testing failure scenarios
):
    print(client.api._prompt_cache.keys())
    try:
        if should_fail:
            original_key = client.api.api_key
            client.api.api_key = "invalid_key"
            try:
                prompt_from_cache = client.api.get_prompt(
                    id=id, name=name, version=version
                )
                return {
                    "success": True,
                    "prompt_id": prompt_from_cache.id,
                    "cache_hit": True
                }
            finally:
                client.api.api_key = original_key
        else:
            prompt = client.api.get_prompt(id=id, name=name, version=version)
        return {
            "success": True,
            "prompt_id": prompt.id,
            "cache_hit": False
        }
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

Install dependencies

pip install fastapi uvicorn

Run test server

uvicorn test_cache:app --reload

Test normal flow

curl "http://localhost:8000/prompt?name=example_prompt"

Test cache with failure

First call to populate cache

curl "http://localhost:8000/prompt?name=example_prompt"

Second call with simulated failure

curl "http://localhost:8000/prompt?name=example_prompt&should_fail=true"

Test with ID

curl "http://localhost:8000/prompt?id=prompt_123"

Test with name and version

curl "http://localhost:8000/prompt?name=example_prompt&version=1"

[willydouhard]

you could move the cache logic in the get_prompt_helper to avoid duplicating it for the sync/async versions.

[willydouhard]

I don't see the modification for the async version of make_gql_call

[Matthieu-OD]

It is lines 1519 and 1532 in the same file

[willydouhard]

Looks promising! I would love to see a test for it.

[Matthieu-OD]

After discussing with @clementsirieix, I'm changing the implementation to avoid a too specific or complex solution. I'm also increasing the timeout to 2 sec instead of 1.