You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/operate/rc/langcache/_index.md
+23Lines changed: 23 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -32,6 +32,29 @@ Using LangCache as a semantic caching service in Redis Cloud has the following b
32
32
-**Simpler Deployments**: Access our managed service via a REST API with automated embedding generation, configurable controls.
33
33
-**Advanced cache management**: Manage data access and privacy, eviction protocols, and monitor usage and cache hit rates.
34
34
35
+
### LLM Cost reduction with LangCache
36
+
37
+
LangCache reduces your LLM costs by caching responses and avoiding repeated API calls. When a response is served from cache, you don’t pay for output tokens. Input token costs are typically offset by embedding and storage costs.
38
+
39
+
For every cached response, you'll save the output token cost. To calculate your monthly savings with LangCache, you can use the following formula:
40
+
41
+
```bash
42
+
Estimated monthly savings with LangCache = (Monthly output token costs) × (Cache hit rate)
43
+
```
44
+
45
+
The more requests you serve from LangCache, the more you save, because you’re not paying to regenerate the output.
46
+
47
+
Here’s an example:
48
+
- Monthly LLM spend: $200
49
+
- Percentage of output tokens in your spend: 60%
50
+
- Cost of output tokens: $200 × 60% = $120
51
+
- Cache hit rate: 50%
52
+
- Estimated savings: $120 × 50% = $60/month
53
+
54
+
{{<note>}}
55
+
The forumla and numbers above will provide a rough estimate of your monthly savings. Actual savings will vary depending on your usage.
56
+
{{</note>}}
57
+
35
58
## LangCache architecture
36
59
37
60
The following diagram displays how you can integrate LangCache into your GenAI app:
0 commit comments