Skip to content

Commit bc13428

Browse files
subtleGradientsubtleGradient
committed
Add prompt-caching examples for Effect AI
- Add typescript/effect-ai/src/prompt-caching/user-message-cache.ts - Demonstrates cache_control using options.openrouter.cacheControl in Prompt - Shows Effect.gen pattern with Layer-based dependency injection - Critical configuration: stream_options.include_usage in model config layer - Evidence-based verification via response.usage.cachedInputTokens
1 parent 6d53a85 commit bc13428

File tree

9 files changed

+559
-1
lines changed

9 files changed

+559
-1
lines changed

‎docs/prompt-caching.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,3 +196,8 @@ See ecosystem-specific examples:
196196
- [user-message-cache.ts](../typescript/ai-sdk-v5/src/prompt-caching/user-message-cache.ts)
197197
- [multi-message-cache.ts](../typescript/ai-sdk-v5/src/prompt-caching/multi-message-cache.ts)
198198
- [no-cache-control.ts](../typescript/ai-sdk-v5/src/prompt-caching/no-cache-control.ts) (control)
199+
200+
- **Effect AI** (@effect/ai): [typescript/effect-ai/src/prompt-caching/](../typescript/effect-ai/src/prompt-caching/)
201+
- [user-message-cache.ts](../typescript/effect-ai/src/prompt-caching/user-message-cache.ts)
202+
- [multi-message-cache.ts](../typescript/effect-ai/src/prompt-caching/multi-message-cache.ts)
203+
- [no-cache-control.ts](../typescript/effect-ai/src/prompt-caching/no-cache-control.ts) (control)

‎typescript/effect-ai/README.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Effect-TS AI Examples
2+
3+
Examples using Effect-TS with @effect/ai and @effect/ai-openrouter for type-safe, composable AI operations.
4+
5+
## Prerequisites
6+
7+
- Bun runtime: `curl -fsSL https://bun.sh/install | bash`
8+
- `OPENROUTER_API_KEY` environment variable
9+
10+
## Running Examples
11+
12+
```bash
13+
# From monorepo root (typescript/)
14+
bun examples
15+
16+
# Or from this workspace
17+
cd effect-ai
18+
bun examples
19+
```
20+
21+
## Features
22+
23+
- [prompt-caching.ts](./src/prompt-caching.ts) - Anthropic caching with Effect patterns
24+
25+
### Key Configuration
26+
27+
**CRITICAL**: The Effect AI example requires:
28+
```typescript
29+
config: {
30+
stream_options: { include_usage: true }
31+
}
32+
```
33+
34+
Without this, `usage.cachedInputTokens` will be undefined in the response.
35+
36+
### Effect Patterns Demonstrated
37+
38+
- `Effect.gen` for generator-based composition
39+
- Layer-based dependency injection
40+
- Type-safe error handling
41+
- Evidence-based validation
42+
43+
## Dependencies
44+
45+
- `@openrouter-examples/shared` - Shared constants (LARGE_SYSTEM_PROMPT) and types
46+
- `@effect/ai` - Effect AI abstractions
47+
- `@effect/ai-openrouter` - OpenRouter provider for Effect AI
48+
- `effect` - Effect-TS core library

‎typescript/effect-ai/package.json‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{
2+
"name": "@openrouter-examples/effect-ai",
3+
"version": "1.0.0",
4+
"private": true,
5+
"type": "module",
6+
"scripts": {
7+
"examples": "bun run src/prompt-caching/user-message-cache.ts && bun run src/prompt-caching/multi-message-cache.ts && bun run src/prompt-caching/no-cache-control.ts",
8+
"typecheck": "tsc --noEmit"
9+
},
10+
"dependencies": {
11+
"@openrouter-examples/shared": "workspace:*",
12+
"@effect/ai": "^0.32.1",
13+
"@effect/ai-openrouter": "^0.6.0",
14+
"@effect/platform": "^0.93.0",
15+
"@effect/platform-bun": "^0.83.0",
16+
"effect": "^3.19.3"
17+
},
18+
"devDependencies": {
19+
"@types/bun": "latest"
20+
}
21+
}

‎typescript/effect-ai/src/prompt-caching/README.md‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# Anthropic Prompt Caching Examples (Effect AI)
2+
3+
This directory contains examples demonstrating Anthropic's prompt caching feature via OpenRouter using @effect/ai and @effect/ai-openrouter.
4+
5+
## What is Prompt Caching?
6+
7+
Anthropic's prompt caching allows you to cache large portions of your prompts to:
8+
- **Reduce costs** - Cached tokens cost significantly less
9+
- **Improve latency** - Cached content is processed faster
10+
- **Enable larger contexts** - Use more context without proportional cost increases
11+
12+
Cache TTL: 5 minutes for ephemeral caches
13+
14+
## Examples
15+
16+
### User Message Cache (`user-message-cache.ts`)
17+
Cache large context in user messages using Effect AI:
18+
```bash
19+
bun run typescript/effect-ai/src/prompt-caching/user-message-cache.ts
20+
```
21+
22+
**Pattern**: User message with `options.openrouter.cacheControl` using Effect.gen
23+
24+
## How to Use with Effect AI
25+
26+
```typescript
27+
import * as OpenRouterClient from '@effect/ai-openrouter/OpenRouterClient';
28+
import * as OpenRouterLanguageModel from '@effect/ai-openrouter/OpenRouterLanguageModel';
29+
import * as LanguageModel from '@effect/ai/LanguageModel';
30+
import * as Prompt from '@effect/ai/Prompt';
31+
import { Effect, Layer, Redacted } from 'effect';
32+
33+
// Create OpenRouter client layer
34+
const OpenRouterClientLayer = OpenRouterClient.layer({
35+
apiKey: Redacted.make(process.env.OPENROUTER_API_KEY!),
36+
}).pipe(Layer.provide(FetchHttpClient.layer));
37+
38+
// Create language model layer with CRITICAL stream_options config
39+
const OpenRouterModelLayer = OpenRouterLanguageModel.layer({
40+
model: 'anthropic/claude-3.5-sonnet',
41+
config: {
42+
stream_options: { include_usage: true }, // CRITICAL: Required!
43+
},
44+
}).pipe(Layer.provide(OpenRouterClientLayer));
45+
46+
// Use in Effect.gen program
47+
const program = Effect.gen(function* () {
48+
const response = yield* LanguageModel.generateText({
49+
prompt: Prompt.make([
50+
{
51+
role: 'user',
52+
content: [
53+
{
54+
type: 'text',
55+
text: 'Large context here...',
56+
options: {
57+
openrouter: {
58+
cacheControl: { type: 'ephemeral' }, // Cache this block
59+
},
60+
},
61+
},
62+
{
63+
type: 'text',
64+
text: 'Your question here',
65+
},
66+
],
67+
},
68+
]),
69+
});
70+
71+
// Check cache metrics
72+
const cachedTokens = response.usage.cachedInputTokens ?? 0;
73+
});
74+
75+
// Run with dependencies
76+
await program.pipe(
77+
Effect.provide(OpenRouterModelLayer),
78+
Effect.runPromise,
79+
);
80+
```
81+
82+
## Important Notes
83+
84+
### Critical Configuration
85+
**MUST include `stream_options: { include_usage: true }` in model config**
86+
- Without this, usage.cachedInputTokens will be undefined
87+
- OpenRouterClient only sets this for streaming by default
88+
- Must be set explicitly in the layer configuration
89+
90+
### Cache Metrics Location
91+
Cache metrics are in `response.usage`:
92+
```typescript
93+
{
94+
inputTokens: number,
95+
outputTokens: number,
96+
cachedInputTokens: number // Number of tokens read from cache
97+
}
98+
```
99+
100+
### Requirements
101+
1. **stream_options.include_usage = true** - In model config layer
102+
2. **Minimum 2048+ tokens** - Smaller content may not be cached
103+
3. **options.openrouter.cacheControl** - On content items in Prompt
104+
4. **Exact match** - Cache only hits on identical content
105+
106+
### Expected Behavior
107+
- **First call**: `cachedInputTokens = 0` (cache miss, creates cache)
108+
- **Second call**: `cachedInputTokens > 0` (cache hit, reads from cache)
109+
110+
### Effect-Specific Patterns
111+
- Use `Effect.gen` for composable effect workflows
112+
- Layer-based dependency injection for client and model
113+
- Type-safe error handling via Effect type
114+
- Structured concurrency with Effect.sleep for delays
115+
116+
## Scientific Method
117+
All examples follow evidence-based verification:
118+
- **Hypothesis**: options.openrouter.cacheControl triggers caching
119+
- **Experiment**: Make identical calls twice
120+
- **Evidence**: Measure via response.usage.cachedInputTokens
121+
- **Analysis**: Compare cache miss vs cache hit

‎typescript/effect-ai/src/prompt-caching/multi-message-cache.ts‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
/**
2+
* Example: Anthropic Prompt Caching - Multi-Message Conversation (Effect AI)
3+
*
4+
* This example demonstrates Anthropic prompt caching in a multi-message conversation
5+
* via OpenRouter using Effect AI.
6+
*
7+
* Pattern: User message cache in multi-turn conversation using Effect patterns
8+
*
9+
* To run: bun run typescript/effect-ai/src/prompt-caching/multi-message-cache.ts
10+
*/
11+
12+
import * as OpenRouterClient from '@effect/ai-openrouter/OpenRouterClient';
13+
import * as OpenRouterLanguageModel from '@effect/ai-openrouter/OpenRouterLanguageModel';
14+
import * as LanguageModel from '@effect/ai/LanguageModel';
15+
import * as Prompt from '@effect/ai/Prompt';
16+
import { FetchHttpClient } from '@effect/platform';
17+
import * as BunContext from '@effect/platform-bun/BunContext';
18+
import { LARGE_SYSTEM_PROMPT } from '@openrouter-examples/shared/constants';
19+
import { Console, Effect, Layer, Redacted } from 'effect';
20+
21+
const program = Effect.gen(function* () {
22+
const testId = Date.now();
23+
const largeContext = `Test ${testId}: Context:\n\n${LARGE_SYSTEM_PROMPT}`;
24+
25+
yield* Console.log('╔════════════════════════════════════════════════════════════════════════════╗');
26+
yield* Console.log('║ Anthropic Prompt Caching - Multi-Message (Effect AI) ║');
27+
yield* Console.log('╚════════════════════════════════════════════════════════════════════════════╝');
28+
yield* Console.log('');
29+
yield* Console.log('Testing cache_control in multi-turn conversation');
30+
yield* Console.log('');
31+
32+
const makePrompt = () =>
33+
Prompt.make([
34+
{
35+
role: 'user' as const,
36+
content: [
37+
{
38+
type: 'text' as const,
39+
text: largeContext,
40+
options: {
41+
openrouter: {
42+
cacheControl: { type: 'ephemeral' as const },
43+
},
44+
},
45+
},
46+
{
47+
type: 'text' as const,
48+
text: "Hello, what's your purpose?",
49+
},
50+
],
51+
},
52+
{
53+
role: 'assistant' as const,
54+
content: "I'm an AI assistant designed to help with various tasks.",
55+
},
56+
{
57+
role: 'user' as const,
58+
content: 'What programming languages do you know?',
59+
},
60+
]);
61+
62+
yield* Console.log('First Call (Cache Miss Expected)');
63+
const response1 = yield* LanguageModel.generateText({
64+
prompt: makePrompt(),
65+
});
66+
const cached1 = response1.usage.cachedInputTokens ?? 0;
67+
yield* Console.log(` Response: ${response1.text.substring(0, 80)}...`);
68+
yield* Console.log(` cached_tokens=${cached1}`);
69+
70+
yield* Effect.sleep('1 second');
71+
72+
yield* Console.log('\nSecond Call (Cache Hit Expected)');
73+
const response2 = yield* LanguageModel.generateText({
74+
prompt: makePrompt(),
75+
});
76+
const cached2 = response2.usage.cachedInputTokens ?? 0;
77+
yield* Console.log(` Response: ${response2.text.substring(0, 80)}...`);
78+
yield* Console.log(` cached_tokens=${cached2}`);
79+
80+
// Analysis
81+
yield* Console.log('\n' + '='.repeat(80));
82+
yield* Console.log('ANALYSIS');
83+
yield* Console.log('='.repeat(80));
84+
yield* Console.log(`First call: cached_tokens=${cached1} (expected: 0)`);
85+
yield* Console.log(`Second call: cached_tokens=${cached2} (expected: >0)`);
86+
87+
const success = cached1 === 0 && cached2 > 0;
88+
89+
if (success) {
90+
yield* Console.log('\n✓ SUCCESS - Multi-message caching is working correctly');
91+
} else {
92+
yield* Console.log('\n✗ FAILURE - Multi-message caching is not working as expected');
93+
}
94+
95+
yield* Console.log('='.repeat(80));
96+
});
97+
98+
const OpenRouterClientLayer = OpenRouterClient.layer({
99+
apiKey: Redacted.make(process.env.OPENROUTER_API_KEY!),
100+
}).pipe(Layer.provide(FetchHttpClient.layer));
101+
102+
const OpenRouterModelLayer = OpenRouterLanguageModel.layer({
103+
model: 'anthropic/claude-3.5-sonnet',
104+
config: {
105+
stream_options: { include_usage: true },
106+
},
107+
}).pipe(Layer.provide(OpenRouterClientLayer));
108+
109+
await program.pipe(
110+
Effect.provide(OpenRouterModelLayer),
111+
Effect.provide(BunContext.layer),
112+
Effect.runPromise,
113+
);
114+
115+
console.log('\n✓ Program completed successfully');

0 commit comments

Comments
 (0)