Skip to content

Commit 7d08f38

Browse files
git-bruhgit-bruh
authored
realtime: Add agents page (#24688)
* [Realtime] Add realtime agents page * add more instructions
1 parent df3a19b commit 7d08f38

File tree

4 files changed

+279
-0
lines changed

4 files changed

+279
-0
lines changed

‎public/images/realtime-agents/create-meeting.png‎

559 KB
Loading

‎public/images/realtime-agents/join-meeting.png‎

100 KB
Loading

‎src/content/docs/realtime/agents/getting-started.mdx‎

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
---
2+
title: Getting started
3+
pcx_content_type: get-started
4+
sidebar:
5+
order: 1
6+
head:
7+
- tag: title
8+
content: Get started - Workers and Wrangler
9+
description: Deploy your first Realtime Agent using the CLI.
10+
---
11+
12+
import { Render, PackageManagers, WranglerConfig, TypeScriptExample } from "~/components";
13+
14+
This guide will instruct you through setting up and deploying your first Realtime Agents project. You will use [Workers](/workers/), the Realtime Agents SDK, a Workers AI binding, and a large language model (LLM) to deploy your first AI-powered application on the Cloudflare global network.
15+
16+
<Render file="prereqs" product="workers" />
17+
18+
## 1. Create a Worker project
19+
20+
You will create a new Worker project using the `create-cloudflare` CLI (C3). [C3](https://github.com/cloudflare/workers-sdk/tree/main/packages/create-cloudflare) is a command-line tool designed to help you set up and deploy new applications to Cloudflare.
21+
22+
Create a new project named `hello-agent` by running:
23+
24+
<PackageManagers type="create" pkg="cloudflare@latest" args={"hello-agent"} />
25+
26+
Running `npm create cloudflare@latest` will prompt you to install the [`create-cloudflare` package](https://www.npmjs.com/package/create-cloudflare), and lead you through setup. C3 will also install [Wrangler](/workers/wrangler/), the Cloudflare Developer Platform CLI.
27+
28+
<Render
29+
file="c3-post-run-steps"
30+
product="workers"
31+
params={{
32+
category: "hello-world",
33+
type: "Worker only",
34+
lang: "TypeScript",
35+
}}
36+
/>
37+
38+
This will create a new `hello-agent` directory. Your new `hello-agent` directory will include:
39+
40+
- A `"Hello World"` [Worker](/workers/get-started/guide/#3-write-code) at `src/index.ts`.
41+
- A [`wrangler.jsonc`](/workers/wrangler/configuration/) configuration file.
42+
43+
Go to your application directory:
44+
45+
```sh
46+
cd hello-agent
47+
```
48+
49+
## 2. Install the Realtime Agents SDK
50+
51+
```sh
52+
npm i @cloudflare/realtime-agents
53+
```
54+
55+
## 3. Connect your Worker to Workers AI
56+
57+
You must create an AI binding for your Worker to connect to Workers AI. [Bindings](/workers/runtime-apis/bindings/) allow your Workers to interact with resources, like Workers AI, on the Cloudflare Developer Platform.
58+
59+
To bind Workers AI to your Worker, add the following to the end of your Wrangler file:
60+
61+
<WranglerConfig>
62+
63+
```toml
64+
[ai]
65+
binding = "AI"
66+
```
67+
68+
</WranglerConfig>
69+
70+
Your binding is [available in your Worker code](/workers/reference/migrate-to-module-workers/#bindings-in-es-modules-format) on [`env.AI`](/workers/runtime-apis/handlers/fetch/).
71+
72+
## 4. Implement the Worker
73+
74+
Update the `index.ts` file in your `hello-agent` application directory with the following code:
75+
76+
<TypeScriptExample filename="index.ts">
77+
78+
```ts
79+
import { DeepgramSTT, TextComponent, RealtimeKitTransport, ElevenLabsTTS, RealtimeAgent } from '@cloudflare/realtime-agents';
80+
81+
class MyTextProcessor extends TextComponent {
82+
env: Env;
83+
84+
constructor(env: Env) {
85+
super();
86+
this.env = env;
87+
}
88+
89+
async onTranscript(text: string, reply: (text: string) => void) {
90+
const { response } = await this.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
91+
prompt: text,
92+
});
93+
reply(response!);
94+
}
95+
}
96+
97+
export class MyAgent extends RealtimeAgent<Env> {
98+
constructor(ctx: DurableObjectState, env: Env) {
99+
super(ctx, env);
100+
}
101+
102+
async init(agentId: string, meetingId: string, authToken: string, workerUrl: string, accountId: string, apiToken: string) {
103+
// Construct your text processor for generating responses to text
104+
const textProcessor = new MyTextProcessor(this.env);
105+
// Construct a Meeting object to join the RTK meeting
106+
const rtkTransport = new RealtimeKitTransport(meetingId, authToken);
107+
108+
// Construct a pipeline to take in meeting audio, transcribe it using
109+
// Deepgram, and pass our generated responses through ElevenLabs to
110+
// be spoken in the meeting
111+
await this.initPipeline(
112+
[
113+
rtkTransport,
114+
new DeepgramSTT(this.env.DEEPGRAM_API_KEY),
115+
textProcessor,
116+
new ElevenLabsTTS(this.env.ELEVENLABS_API_KEY),
117+
rtkTransport,
118+
],
119+
agentId,
120+
workerUrl,
121+
accountId,
122+
apiToken,
123+
);
124+
125+
const { meeting } = rtkTransport;
126+
127+
// The RTK meeting object is accessible to us, so we can register handlers
128+
// on various events like participant joins/leaves, chat, etc.
129+
// This is optional
130+
meeting.participants.joined.on('participantJoined', (participant) => {
131+
textProcessor.speak(`Participant Joined ${participant.name}`);
132+
});
133+
meeting.participants.joined.on('participantLeft', (participant) => {
134+
textProcessor.speak(`Participant Left ${participant.name}`);
135+
});
136+
137+
// Make sure to actually join the meeting after registering all handlers
138+
await meeting.join();
139+
}
140+
141+
async deinit() {
142+
// Add any other cleanup logic required
143+
await this.deinitPipeline();
144+
}
145+
}
146+
147+
export default {
148+
async fetch(request, env, _ctx): Promise<Response> {
149+
const url = new URL(request.url);
150+
const meetingId = url.searchParams.get('meetingId');
151+
if (!meetingId) {
152+
return new Response(null, { status: 400 });
153+
}
154+
155+
const agentId = meetingId;
156+
const agent = env.MY_AGENT.idFromName(meetingId);
157+
const stub = env.MY_AGENT.get(agent);
158+
// The fetch method is implemented for handling internal pipeline logic
159+
if (url.pathname.startsWith('/agentsInternal')) {
160+
return stub.fetch(request);
161+
}
162+
163+
// Your logic continues here
164+
switch (url.pathname) {
165+
case '/init':
166+
// This is the authToken for joining a meeting, it can be passed
167+
// in query parameters as well if needed
168+
const authHeader = request.headers.get('Authorization');
169+
if (!authHeader) {
170+
return new Response(null, { status: 401 });
171+
}
172+
173+
// We just need the part after `Bearer `
174+
await stub.init(agentId, meetingId, authHeader.split(' ')[1], url.host, env.ACCOUNT_ID, env.API_TOKEN);
175+
176+
return new Response(null, { status: 200 });
177+
case '/deinit':
178+
await stub.deinit();
179+
return new Response(null, { status: 200 });
180+
}
181+
182+
return new Response(null, { status: 404 });
183+
},
184+
} satisfies ExportedHandler<Env>;
185+
186+
```
187+
</TypeScriptExample>
188+
189+
The Realtime Agents SDK provides several elements that work together to create an end-to-end pipeline
190+
191+
- `RealtimeKitTransport`: Represents a RealtimeKit meeting that will be joined by the agent
192+
193+
- `DeepgramSTT`: Takes in meeting audio and provides transcripts powered by Deepgram
194+
195+
- `TextComponent`: A concrete implementation for this element needs to be provided by the user as it is responsible for processing the text generated in the meeting and sending back responses. We have implemented it in the `MyTextProcessor` class
196+
197+
- `ElevenLabsTTS`: Converts the generated responses to audio to be spoken in the meeting
198+
199+
We use all of these elements together to create a simple chatbot-like pipeline. As a pre-requisite, we require the meeting ID to be joined along with an authorization token for joining the meeting, which is passed during the worker invocation. Additionally, our class must extend `RealtimeAgent` as it contains certain internal logic to handle interactions with our pipeline backend
200+
201+
In `wrangler.jsonc`, append the following fields to enable the [Node.js Compatibility](https://developers.cloudflare.com/workers/runtime-apis/nodejs/) flag and create our Durable Object:
202+
203+
```json
204+
"compatibility_flags": ["nodejs_compat"],
205+
"migrations": [
206+
{
207+
"new_sqlite_classes": ["MyAgent"],
208+
"tag": "v1",
209+
},
210+
],
211+
"durable_objects": {
212+
"bindings": [
213+
{
214+
"class_name": "MyAgent",
215+
"name": "MY_AGENT",
216+
},
217+
],
218+
},
219+
```
220+
221+
You must also setup a few [secrets](https://developers.cloudflare.com/workers/configuration/secrets/):
222+
223+
- `ACCOUNT_ID`: Your Cloudflare account ID
224+
- `API_TOKEN`: Cloudflare API token scoped for `Admin` access to `Realtime`
225+
- `ELEVENLABS_API_KEY`, `DEEPGRAM_API_KEY`: ElevenLabs & Deepgram API keys
226+
227+
## 5. Deploy your AI Worker
228+
229+
Before deploying your AI Worker globally, log in with your Cloudflare account by running:
230+
231+
```sh
232+
npx wrangler login
233+
```
234+
235+
You will be directed to a web page asking you to log in to the Cloudflare dashboard. After you have logged in, you will be asked if Wrangler can make changes to your Cloudflare account. Scroll down and select **Allow** to continue.
236+
237+
Finally, deploy your Worker to make your project accessible on the Internet. To deploy your Worker, run:
238+
239+
```sh
240+
npx wrangler deploy
241+
```
242+
243+
```sh output
244+
https://hello-agent.<YOUR_SUBDOMAIN>.workers.dev
245+
```
246+
247+
## 6. Generate a RealtimeKit token
248+
249+
Finally, to invoke the worker, we need to generate a RealtimeKit token from the [dashboard](https://dash.realtime.cloudflare.com/dashboard):
250+
251+
1. Go to the `Meetings` tab and click on `Create Meeting`:
252+
253+
![Meetings Tab](public/images/realtime-agents/create-meeting.png)
254+
255+
2. Click on `Join` next to the meeting and generate the RealtimeKit link. This contains the `meetingId` (`bbbb2fac-953c-4239-9ba8-75ba912d76fc`) and the `authToken` to be passed in the final step:
256+
257+
`https://demo.realtime.cloudflare.com/v2/meeting?id=bbbb2fac-953c-4239-9ba8-75ba912d76fc&authToken=ey...`
258+
259+
![Join Flow](public/images/realtime-agents/join-meeting.png)
260+
261+
3. Repeat the same `Join` flow to join the meeting yourself before adding in the Agent
262+
263+
Finally, invoke the worker to make the agent join a meeting:
264+
265+
```sh
266+
curl -X POST https://hello-agent.<YOUR_SUBDOMAIN>.workers.dev/init?meetingId=<REALTIME_KIT_MEETING_ID> -H "Authorization: Bearer <REALTIME_KIT_AUTH_TOKEN>"
267+
```
268+
269+
## Related resources
270+
271+
- [Cloudflare Developers community on Discord](https://discord.cloudflare.com) - Submit feature requests, report bugs, and share your feedback directly with the Cloudflare team by joining the Cloudflare Discord server.

‎src/content/docs/realtime/agents/index.mdx‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
title: Realtime Agents
3+
pcx_content_type: navigation
4+
sidebar:
5+
group:
6+
hideIndex: true
7+
order: 5
8+
---

0 commit comments

Comments
 (0)