[v0.1] Semantic Cache: Production-ready semantic caching

[rootfs]

Acceptance

• Support more backends in semantic caching
• Hit rates are tracked; cache eviction is configurable
• Performance benchmarks are included.

[aeft]

There are several problems we need to solve to provide useful semantic caching:

1. Currently, we only use the most recent user message as the query. This may cause the cache to be unhelpful (or even completely incorrect) in some context-sensitive conversations.
2. The cache may be disabled for certain types of queries. Take mathematical calculations as an example, where two queries that differ by only one number can produce completely different results. Lifetimes should be associated with individual cache entries #160
3. Evaluate the correctness of cached results: how well do cached results match the outputs generated by the LLM each time? We may use 33K Chatbot Arena Conversation Data. We may also consider the trade-off between correctness and token-saving.

[sairampillai]

There are several problems we need to solve to provide useful semantic caching:

1. Currently, we only use the most recent user message as the query. This may cause the cache to be unhelpful (or even completely incorrect) in some context-sensitive conversations.
2. The cache may be disabled for certain types of queries. Take mathematical calculations as an example, where two queries that differ by only one number can produce completely different results. Lifetimes should be associated with individual cache entries #160
3. Evaluate the correctness of cached results: how well do cached results match the outputs generated by the LLM each time? We may use 33K Chatbot Arena Conversation Data. We may also consider the trade-off between correctness and token-saving.

@aeft Can we split these into it's own issues and design/implement individually? I can help solving these problems.