Skip to content

Commit 6d53a85

Browse files
subtleGradientsubtleGradient
committed
Add prompt-caching examples for AI SDK v5
- Add typescript/ai-sdk-v5/src/prompt-caching/user-message-cache.ts - Demonstrates cache_control using providerOptions.openrouter.cacheControl - Shows critical configuration: extraBody.stream_options.include_usage - Evidence-based verification via providerMetadata.openrouter.usage
1 parent 314435b commit 6d53a85

File tree

9 files changed

+602
-1
lines changed

9 files changed

+602
-1
lines changed

‎docs/prompt-caching.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,3 +191,8 @@ See ecosystem-specific examples:
191191
- [user-message-cache.ts](../typescript/fetch/src/prompt-caching/user-message-cache.ts)
192192
- [multi-message-cache.ts](../typescript/fetch/src/prompt-caching/multi-message-cache.ts)
193193
- [no-cache-control.ts](../typescript/fetch/src/prompt-caching/no-cache-control.ts) (control)
194+
195+
- **AI SDK v5** (Vercel): [typescript/ai-sdk-v5/src/prompt-caching/](../typescript/ai-sdk-v5/src/prompt-caching/)
196+
- [user-message-cache.ts](../typescript/ai-sdk-v5/src/prompt-caching/user-message-cache.ts)
197+
- [multi-message-cache.ts](../typescript/ai-sdk-v5/src/prompt-caching/multi-message-cache.ts)
198+
- [no-cache-control.ts](../typescript/ai-sdk-v5/src/prompt-caching/no-cache-control.ts) (control)

‎typescript/ai-sdk-v5/README.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# AI SDK v5 Examples
2+
3+
Examples using Vercel AI SDK v5 with @openrouter/ai-sdk-provider.
4+
5+
## Prerequisites
6+
7+
- Bun runtime: `curl -fsSL https://bun.sh/install | bash`
8+
- `OPENROUTER_API_KEY` environment variable
9+
10+
## Running Examples
11+
12+
```bash
13+
# From monorepo root (typescript/)
14+
bun examples
15+
16+
# Or from this workspace
17+
cd ai-sdk-v5
18+
bun examples
19+
```
20+
21+
## Features
22+
23+
- [prompt-caching.ts](./src/prompt-caching.ts) - Anthropic caching with AI SDK v5
24+
25+
### Key Configuration
26+
27+
**CRITICAL**: The AI SDK example requires:
28+
```typescript
29+
extraBody: {
30+
stream_options: { include_usage: true }
31+
}
32+
```
33+
34+
Without this, usage details (including cached_tokens) are not populated in the response.
35+
36+
## Dependencies
37+
38+
- `@openrouter-examples/shared` - Shared constants (LARGE_SYSTEM_PROMPT) and types
39+
- `@openrouter/ai-sdk-provider` - OpenRouter provider for AI SDK
40+
- `ai` v5.x - Vercel AI SDK

‎typescript/ai-sdk-v5/package.json‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"name": "@openrouter-examples/ai-sdk-v5",
3+
"version": "1.0.0",
4+
"private": true,
5+
"type": "module",
6+
"scripts": {
7+
"examples": "bun run src/prompt-caching/user-message-cache.ts && bun run src/prompt-caching/multi-message-cache.ts && bun run src/prompt-caching/no-cache-control.ts",
8+
"typecheck": "tsc --noEmit"
9+
},
10+
"dependencies": {
11+
"@openrouter-examples/shared": "workspace:*",
12+
"@openrouter/ai-sdk-provider": "^1.2.1",
13+
"ai": "^5.0.92"
14+
},
15+
"devDependencies": {
16+
"@types/bun": "latest"
17+
}
18+
}

‎typescript/ai-sdk-v5/src/prompt-caching/README.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Anthropic Prompt Caching Examples (AI SDK v5)
2+
3+
This directory contains examples demonstrating Anthropic's prompt caching feature via OpenRouter using Vercel AI SDK v5.
4+
5+
## What is Prompt Caching?
6+
7+
Anthropic's prompt caching allows you to cache large portions of your prompts to:
8+
- **Reduce costs** - Cached tokens cost significantly less
9+
- **Improve latency** - Cached content is processed faster
10+
- **Enable larger contexts** - Use more context without proportional cost increases
11+
12+
Cache TTL: 5 minutes for ephemeral caches
13+
14+
## Examples
15+
16+
### User Message Cache (`user-message-cache.ts`)
17+
Cache large context in user messages using AI SDK:
18+
```bash
19+
bun run typescript/ai-sdk-v5/src/prompt-caching/user-message-cache.ts
20+
```
21+
22+
**Pattern**: User message with `providerOptions.openrouter.cacheControl`
23+
24+
## How to Use with AI SDK
25+
26+
```typescript
27+
import { createOpenRouter } from '@openrouter/ai-sdk-provider';
28+
import { generateText } from 'ai';
29+
30+
// CRITICAL: Must include stream_options for usage details
31+
const openrouter = createOpenRouter({
32+
apiKey: process.env.OPENROUTER_API_KEY,
33+
extraBody: {
34+
stream_options: { include_usage: true }, // Required!
35+
},
36+
});
37+
38+
const result = await generateText({
39+
model: openrouter('anthropic/claude-3.5-sonnet'),
40+
messages: [
41+
{
42+
role: 'user',
43+
content: [
44+
{
45+
type: 'text',
46+
text: 'Large context here...',
47+
providerOptions: {
48+
openrouter: {
49+
cacheControl: { type: 'ephemeral' }, // Cache this block
50+
},
51+
},
52+
},
53+
{
54+
type: 'text',
55+
text: 'Your question here',
56+
},
57+
],
58+
},
59+
],
60+
});
61+
62+
// Check cache metrics
63+
const cachedTokens = result.providerMetadata?.openrouter?.usage?.promptTokensDetails?.cachedTokens ?? 0;
64+
```
65+
66+
## Important Notes
67+
68+
### Critical Configuration
69+
**MUST include `extraBody: { stream_options: { include_usage: true } }`**
70+
- Without this, usage details (including cached_tokens) are not populated
71+
- This is a provider-level configuration, not per-request
72+
73+
### Cache Metrics Location
74+
Cache metrics are in `providerMetadata.openrouter.usage`:
75+
```typescript
76+
{
77+
promptTokens: number,
78+
completionTokens: number,
79+
promptTokensDetails: {
80+
cachedTokens: number // Number of tokens read from cache
81+
}
82+
}
83+
```
84+
85+
### Requirements
86+
1. **stream_options.include_usage = true** - CRITICAL for usage details
87+
2. **Minimum 2048+ tokens** - Smaller content may not be cached
88+
3. **providerOptions.openrouter.cacheControl** - On content items, not messages
89+
4. **Exact match** - Cache only hits on identical content
90+
91+
### Expected Behavior
92+
- **First call**: `cachedTokens = 0` (cache miss, creates cache)
93+
- **Second call**: `cachedTokens > 0` (cache hit, reads from cache)
94+
95+
## Scientific Method
96+
All examples follow evidence-based verification:
97+
- **Hypothesis**: providerOptions.openrouter.cacheControl triggers caching
98+
- **Experiment**: Make identical calls twice
99+
- **Evidence**: Measure via providerMetadata.openrouter.usage
100+
- **Analysis**: Compare cache miss vs cache hit

‎typescript/ai-sdk-v5/src/prompt-caching/multi-message-cache.ts‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
/**
2+
* Example: Anthropic Prompt Caching - Multi-Message Conversation (AI SDK v5)
3+
*
4+
* This example demonstrates Anthropic prompt caching in a multi-message conversation
5+
* via OpenRouter using Vercel AI SDK v5.
6+
*
7+
* Pattern: User message cache in multi-turn conversation
8+
* - Cache large context in first user message
9+
* - Cache persists through conversation history
10+
*
11+
* To run: bun run typescript/ai-sdk-v5/src/prompt-caching/multi-message-cache.ts
12+
*/
13+
14+
import { LARGE_SYSTEM_PROMPT } from '@openrouter-examples/shared/constants';
15+
import { createOpenRouter } from '@openrouter/ai-sdk-provider';
16+
import { generateText } from 'ai';
17+
18+
const openrouter = createOpenRouter({
19+
apiKey: process.env.OPENROUTER_API_KEY,
20+
extraBody: {
21+
stream_options: { include_usage: true },
22+
},
23+
});
24+
25+
async function main() {
26+
console.log('╔════════════════════════════════════════════════════════════════════════════╗');
27+
console.log('║ Anthropic Prompt Caching - Multi-Message (AI SDK v5) ║');
28+
console.log('╚════════════════════════════════════════════════════════════════════════════╝');
29+
console.log();
30+
console.log('Testing cache_control in multi-turn conversation');
31+
console.log();
32+
33+
try {
34+
const testId = Date.now();
35+
const model = openrouter('anthropic/claude-3-5-sonnet');
36+
const largeContext = `Test ${testId}: Context:\n\n${LARGE_SYSTEM_PROMPT}`;
37+
38+
// First call with conversation history
39+
console.log('First Call (Cache Miss Expected)');
40+
const result1 = await generateText({
41+
model,
42+
messages: [
43+
{
44+
role: 'user',
45+
content: [
46+
{
47+
type: 'text',
48+
text: largeContext,
49+
providerOptions: {
50+
openrouter: {
51+
cacheControl: { type: 'ephemeral' },
52+
},
53+
},
54+
},
55+
{
56+
type: 'text',
57+
text: "Hello, what's your purpose?",
58+
},
59+
],
60+
},
61+
{
62+
role: 'assistant',
63+
content: "I'm an AI assistant designed to help with various tasks.",
64+
},
65+
{
66+
role: 'user',
67+
content: 'What programming languages do you know?',
68+
},
69+
],
70+
});
71+
72+
const cached1 = result1.providerMetadata?.openrouter?.usage?.promptTokensDetails?.cachedTokens ?? 0;
73+
console.log(` Response: ${result1.text.substring(0, 80)}...`);
74+
console.log(` cached_tokens=${cached1}`);
75+
76+
await new Promise((resolve) => setTimeout(resolve, 1000));
77+
78+
// Second identical call - should hit cache
79+
console.log('\nSecond Call (Cache Hit Expected)');
80+
const result2 = await generateText({
81+
model,
82+
messages: [
83+
{
84+
role: 'user',
85+
content: [
86+
{
87+
type: 'text',
88+
text: largeContext,
89+
providerOptions: {
90+
openrouter: {
91+
cacheControl: { type: 'ephemeral' },
92+
},
93+
},
94+
},
95+
{
96+
type: 'text',
97+
text: "Hello, what's your purpose?",
98+
},
99+
],
100+
},
101+
{
102+
role: 'assistant',
103+
content: "I'm an AI assistant designed to help with various tasks.",
104+
},
105+
{
106+
role: 'user',
107+
content: 'What programming languages do you know?',
108+
},
109+
],
110+
});
111+
112+
const cached2 = result2.providerMetadata?.openrouter?.usage?.promptTokensDetails?.cachedTokens ?? 0;
113+
console.log(` Response: ${result2.text.substring(0, 80)}...`);
114+
console.log(` cached_tokens=${cached2}`);
115+
116+
// Analysis
117+
console.log('\n' + '='.repeat(80));
118+
console.log('ANALYSIS');
119+
console.log('='.repeat(80));
120+
console.log(`First call: cached_tokens=${cached1} (expected: 0)`);
121+
console.log(`Second call: cached_tokens=${cached2} (expected: >0)`);
122+
123+
const success = cached1 === 0 && cached2 > 0;
124+
console.log(`\nResult: ${success ? '✓ CACHE WORKING' : '✗ CACHE NOT WORKING'}`);
125+
126+
if (success) {
127+
console.log('\n✓ SUCCESS - Multi-message caching is working correctly');
128+
} else {
129+
console.log('\n✗ FAILURE - Multi-message caching is not working as expected');
130+
}
131+
} catch (error) {
132+
console.error('\n❌ ERROR:', error);
133+
process.exit(1);
134+
}
135+
}
136+
137+
main();

0 commit comments

Comments
 (0)