Checking cache on model group fallbacks

[Manouchehri]

What happened?

When a call in an LLM group fails, it seems like the cache is checked again. This is pretty wasteful since it's highly unlikely that this will ever result in a cache hit.

I would assume this is a bug across all caching providers, not just S3.

Relevant log output

No response

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/

[krrishdholakia]

it seems like the cache is checked again

it seems reasonable to check cache when the cache key changes

we have optimizations around reducing the number of cache get requests - e.g. batching gets for a key - https://docs.litellm.ai/docs/proxy/caching#turn-on-batch_redis_requests

--

i don't view this as a bug, but i'm happy to convert to discussion and see what works best!