Skip to content

Commit 01cca3a

Browse files
cmilesbcmilesb
authored
Merge pull request #1839 from redis/revert-1838-revert-1703-DOC-5172
Revert "Revert "RC: LangCache public preview""
2 parents 82858d0 + d2f0773 commit 01cca3a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+1217
-107
lines changed

‎content/develop/ai/langcache.md‎

Lines changed: 0 additions & 96 deletions
This file was deleted.

‎content/develop/ai/langcache/_index.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
Title: Redis LangCache
3+
alwaysopen: false
4+
categories:
5+
- docs
6+
- develop
7+
- ai
8+
description: Store LLM responses for AI apps in a semantic cache.
9+
linkTitle: LangCache
10+
hideListLinks: true
11+
weight: 30
12+
bannerText: LangCache is currently available in preview. Features and behavior are subject to change.
13+
bannerChildren: true
14+
---
15+
16+
Redis LangCache is a fully-managed semantic caching service that reduces large language model (LLM) costs and improves response times for AI applications.
17+
18+
## LangCache overview
19+
20+
LangCache uses semantic caching to store and reuse previous LLM responses for repeated queries. Instead of calling the LLM for every request, LangCache checks if a similar response has already been generated and is stored in the cache. If a match is found, LangCache returns the cached response instantly, saving time and resources.
21+
22+
Imagine you’re using an LLM to build an agent to answer questions about your company's products. Your users may ask questions like the following:
23+
24+
- "What are the features of Product A?"
25+
- "Can you list the main features of Product A?"
26+
- "Tell me about Product A’s features."
27+
28+
These prompts may have slight variations, but they essentially ask the same question. LangCache can help you avoid calling the LLM for each of these prompts by caching the response to the first prompt and returning it for any similar prompts.
29+
30+
Using LangCache as a semantic caching service has the following benefits:
31+
32+
- **Lower LLM costs**: Reduce costly LLM calls by easily storing the most frequently-requested responses.
33+
- **Faster AI app responses**: Get faster AI responses by retrieving previously-stored requests from memory.
34+
- **Simpler Deployments**: Access our managed service using a REST API with automated embedding generation, configurable controls, and no database management required.
35+
- **Advanced cache management**: Manage data access and privacy, eviction protocols, and monitor usage and cache hit rates.
36+
37+
LangCache works well for the following use cases:
38+
39+
- **AI assistants and chatbots**: Optimize conversational AI applications by caching common responses and reducing latency for frequently asked questions.
40+
- **RAG applications**: Enhance retrieval-augmented generation performance by caching responses to similar queries, reducing both cost and response time.
41+
- **AI agents**: Improve multi-step reasoning chains and agent workflows by caching intermediate results and common reasoning patterns.
42+
- **AI gateways**: Integrate LangCache into centralized AI gateway services to manage and control LLM costs across multiple applications..
43+
44+
### LLM cost reduction with LangCache
45+
46+
{{< embed-md "langcache-cost-reduction.md" >}}
47+
48+
## LangCache architecture
49+
50+
The following diagram displays how you can integrate LangCache into your GenAI app:
51+
52+
{{< image filename="images/rc/langcache-process.png" alt="The LangCache process diagram." >}}
53+
54+
1. A user sends a prompt to your AI app.
55+
1. Your app sends the prompt to LangCache through the `POST /v1/caches/{cacheId}/search` endpoint.
56+
1. LangCache calls an embedding model service to generate an embedding for the prompt.
57+
1. LangCache searches the cache to see if a similar response already exists by matching the embeddings of the new query with the stored embeddings.
58+
1. If a semantically similar entry is found (also known as a cache hit), LangCache gets the cached response and returns it to your app. Your app can then send the cached response back to the user.
59+
1. If no match is found (also known as a cache miss), your app receives an empty response from LangCache. Your app then queries your chosen LLM to generate a new response.
60+
1. Your app sends the prompt and the new response to LangCache through the `POST /v1/caches/{cacheId}/entries` endpoint.
61+
1. LangCache stores the embedding with the new response in the cache for future use.
62+
63+
See the [LangCache API reference]({{< relref "/develop/ai/langcache/api-reference" >}}) for more information on how to use the LangCache API.
64+
65+
## Get started
66+
67+
LangCache is currently in preview:
68+
69+
- Public preview on [Redis Cloud]({{< relref "/operate/rc/langcache" >}})
70+
- Fully-managed [private preview](https://redis.io/langcache/)
71+
72+
{{< multitabs id="langcache-get-started"
73+
tab1="Redis Cloud"
74+
tab2="Private preview" >}}
75+
76+
{{< embed-md "rc-langcache-get-started.md" >}}
77+
78+
-tab-sep-
79+
80+
### Prerequisites
81+
82+
To use LangCache in private preview, you need:
83+
84+
- An AI application that makes LLM API calls
85+
- A use case involving repetitive or similar queries
86+
- Willingness to provide feedback during the preview phase
87+
88+
### Access
89+
90+
LangCache is offered as a fully-managed service. During the private preview:
91+
92+
- Participation is free
93+
- Usage limits may apply
94+
- Dedicated support is provided
95+
- Regular feedback sessions are conducted
96+
97+
### Data security and privacy
98+
99+
LangCache stores your data on your Redis servers. Redis does not access your data or use it to train AI models. The service maintains enterprise-grade security and privacy standards.
100+
101+
### Support
102+
103+
Private preview participants receive:
104+
105+
- Dedicated onboarding resources
106+
- Documentation and tutorials
107+
- Email and chat support
108+
- Regular check-ins with the product team
109+
- Exclusive roadmap updates
110+
111+
For more information about joining the private preview, visit the [Redis LangCache website](https://redis.io/langcache/).
112+
113+
{{< /multitabs >}}

‎content/develop/ai/langcache/api-examples.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
---
2+
alwaysopen: false
3+
categories:
4+
- docs
5+
- develop
6+
- ai
7+
description: Learn to use the Redis LangCache API for semantic caching.
8+
hideListLinks: true
9+
linktitle: API and SDK examples
10+
title: Use the LangCache API and SDK
11+
weight: 10
12+
---
13+
14+
Use the LangCache API from your client app to store and retrieve LLM, RAG, or agent responses.
15+
16+
To access the LangCache API, you need:
17+
18+
- LangCache API base URL
19+
- LangCache service API key
20+
- Cache ID
21+
22+
When you call the API, you need to pass the LangCache API key in the `Authorization` header as a Bearer token and the Cache ID as the `cacheId` path parameter.
23+
24+
For example, to search the cache using `cURL`:
25+
26+
```bash
27+
curl -s -X POST "https://$HOST/v1/caches/$CACHE_ID/entires/search" \
28+
-H "accept: application/json" \
29+
-H "Authorization: Bearer $API_KEY" \
30+
-d "{ 'prompt': 'What is semantic caching' }"
31+
```
32+
33+
- The example expects several variables to be set in the shell:
34+
35+
- **$HOST** - the LangCache API base URL
36+
- **$CACHE_ID** - the Cache ID of your cache
37+
- **$API_KEY** - The LangCache API token
38+
39+
{{% info %}}
40+
This example uses `cURL` and Linux shell scripts to demonstrate the API; you can use any standard REST client or library.
41+
{{% /info %}}
42+
43+
You can also use the [LangCache SDKs](#langcache-sdk) for Javascript and Python to access the API.
44+
45+
## API examples
46+
47+
### Search LangCache for similar responses
48+
49+
Use `POST /v1/caches/{cacheId}/entries/search` to search the cache for matching responses to a user prompt.
50+
51+
```sh
52+
POST https://[host]/v1/caches/{cacheId}/entries/search
53+
{
54+
"prompt": "User prompt text"
55+
}
56+
```
57+
58+
Place this call in your client app right before you call your LLM's REST API. If LangCache returns a response, you can send that response back to the user instead of calling the LLM.
59+
60+
If LangCache does not return a response, you should call your LLM's REST API to generate a new response. After you get a response from the LLM, you can [store it in LangCache](#store-a-new-response-in-langcache) for future use.
61+
62+
You can also scope the responses returned from LangCache by adding an `attributes` object to the request. LangCache will only return responses that match the attributes you specify.
63+
64+
```sh
65+
POST https://[host]/v1/caches/{cacheId}/entries/search
66+
{
67+
"prompt": "User prompt text",
68+
"attributes": {
69+
"customAttributeName": "customAttributeValue"
70+
}
71+
}
72+
```
73+
74+
### Store a new response in LangCache
75+
76+
Use `POST /v1/caches/{cacheId}/entries` to store a new response in the cache.
77+
78+
```sh
79+
POST https://[host]/v1/caches/{cacheId}/entries
80+
{
81+
"prompt": "User prompt text",
82+
"response": "LLM response text"
83+
}
84+
```
85+
86+
Place this call in your client app after you get a response from the LLM. This will store the response in the cache for future use.
87+
88+
You can also store the responses with custom attributes by adding an `attributes` object to the request.
89+
90+
```sh
91+
POST https://[host]/v1/caches/{cacheId}/entries
92+
{
93+
"prompt": "User prompt text",
94+
"response": "LLM response text",
95+
"attributes": {
96+
"customAttributeName": "customAttributeValue"
97+
}
98+
}
99+
```
100+
101+
### Delete cached responses
102+
103+
Use `DELETE /v1/caches/{cacheId}/entries/{entryId}` to delete a cached response from the cache.
104+
105+
You can also use `DELETE /v1/caches/{cacheId}/entries` to delete multiple cached responses at once. If you provide an `attributes` object, LangCache will delete all responses that match the attributes you specify.
106+
107+
```sh
108+
DELETE https://[host]/v1/caches/{cacheId}/entries
109+
{
110+
"attributes": {
111+
"customAttributeName": "customAttributeValue"
112+
}
113+
}
114+
```
115+
## LangCache SDK
116+
117+
If your app is written in Javascript or Python, you can also use the LangCache Software Development Kits (SDKs) to access the API.
118+
119+
To learn how to use the LangCache SDKs:
120+
121+
- [LangCache SDK for Javascript](https://www.npmjs.com/package/@redis-ai/langcache)
122+
- [LangCache SDK for Python](https://pypi.org/project/langcache/)

‎content/develop/ai/langcache/api-reference.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
Title: LangCache REST API
3+
linkTitle: API reference
4+
layout: apireference
5+
type: page
6+
params:
7+
sourcefile: ./api.yaml
8+
sortOperationsAlphabetically: false
9+
---

0 commit comments

Comments
 (0)