Skip to content

Commit b06c9d0

Browse files
git-bruhgit-bruh
authored
Create getting-started.mdx
1 parent 3be8098 commit b06c9d0

File tree

1 file changed

+224
-0
lines changed

1 file changed

+224
-0
lines changed

‎src/content/docs/realtime/agents/getting-started.mdx‎

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
---
2+
title: Realtime Agents
3+
pcx_content_type: get-started
4+
sidebar:
5+
order: 1
6+
head:
7+
- tag: title
8+
content: Get started - Workers and Wrangler
9+
description: Deploy your first Realtime Agent using the CLI.
10+
---
11+
12+
import { Render, PackageManagers, WranglerConfig, TypeScriptExample } from "~/components";
13+
14+
This guide will instruct you through setting up and deploying your first Realtime Agents project. You will use [Workers](/workers/), the [Realtime Agents SDK](TODO), a Workers AI binding, and a large language model (LLM) to deploy your first AI-powered application on the Cloudflare global network.
15+
16+
<Render file="prereqs" product="workers" />
17+
18+
## 1. Create a Worker project
19+
20+
You will create a new Worker project using the `create-cloudflare` CLI (C3). [C3](https://github.com/cloudflare/workers-sdk/tree/main/packages/create-cloudflare) is a command-line tool designed to help you set up and deploy new applications to Cloudflare.
21+
22+
Create a new project named `hello-agent` by running:
23+
24+
<PackageManagers type="create" pkg="cloudflare@latest" args={"hello-agent"} />
25+
26+
Running `npm create cloudflare@latest` will prompt you to install the [`create-cloudflare` package](https://www.npmjs.com/package/create-cloudflare), and lead you through setup. C3 will also install [Wrangler](/workers/wrangler/), the Cloudflare Developer Platform CLI.
27+
28+
<Render
29+
file="c3-post-run-steps"
30+
product="workers"
31+
params={{
32+
category: "hello-world",
33+
type: "Worker only",
34+
lang: "TypeScript",
35+
}}
36+
/>
37+
38+
This will create a new `hello-agent` directory. Your new `hello-agent` directory will include:
39+
40+
- A `"Hello World"` [Worker](/workers/get-started/guide/#3-write-code) at `src/index.ts`.
41+
- A [`wrangler.jsonc`](/workers/wrangler/configuration/) configuration file.
42+
43+
Go to your application directory:
44+
45+
```sh
46+
cd hello-agent
47+
```
48+
49+
## 2. Install the Realtime Agents SDK
50+
51+
```sh
52+
npm i @cloudflare/realtime-agents
53+
```
54+
55+
## 3. Connect your Worker to Workers AI
56+
57+
You must create an AI binding for your Worker to connect to Workers AI. [Bindings](/workers/runtime-apis/bindings/) allow your Workers to interact with resources, like Workers AI, on the Cloudflare Developer Platform.
58+
59+
To bind Workers AI to your Worker, add the following to the end of your Wrangler file:
60+
61+
<WranglerConfig>
62+
63+
```toml
64+
[ai]
65+
binding = "AI"
66+
```
67+
68+
</WranglerConfig>
69+
70+
Your binding is [available in your Worker code](/workers/reference/migrate-to-module-workers/#bindings-in-es-modules-format) on [`env.AI`](/workers/runtime-apis/handlers/fetch/).
71+
72+
## 4. Implement the Worker
73+
74+
Update the `index.ts` file in your `hello-agent` application directory with the following code:
75+
76+
<TypeScriptExample filename="index.ts">
77+
78+
```ts
79+
import { DeepgramSTT, TextProcessor, Meeting, ElevenLabsTTS, ArtificialObject } from '@cloudflare/realtime-agents';
80+
81+
class MyTextProcessor extends TextProcessor {
82+
env: Env;
83+
84+
constructor(env: Env) {
85+
super();
86+
this.env = env;
87+
}
88+
89+
async onTranscript(text: string, reply: (text: string) => void) {
90+
const { response } = await this.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
91+
prompt: text,
92+
});
93+
reply(response);
94+
}
95+
}
96+
97+
export class RoomObject extends ArtificialObject<Env> {
98+
constructor(ctx: DurableObjectState, env: Env) {
99+
super(ctx, env);
100+
}
101+
102+
async init(meetingId: string, authToken: string, workerUrl: string) {
103+
// Construct your text processor for generating responses to text
104+
const textProcessor = new MyTextProcessor(this.env);
105+
// Construct a Meeting object to join the RTK meeting
106+
const meeting = await Meeting.init(meetingId, authToken, [
107+
{
108+
media_kind: 'audio',
109+
stream_kind: 'webcam',
110+
preset_name: 'group_call_host',
111+
},
112+
]);
113+
114+
// Construct a pipeline to take in meeting audio, transcribe it using
115+
// Deepgram, and pass our generated responses through ElevenLabs to
116+
// be spoken in the meeting
117+
await this.initPipeline(
118+
[meeting, new DeepgramSTT(this.env.DEEPGRAM_API_KEY), textProcessor, new ElevenLabsTTS(this.env.ELEVENLABS_API_KEY), meeting],
119+
meetingId,
120+
workerUrl,
121+
);
122+
123+
// The RTK meeting object is accessible to us, so we can register handlers
124+
// on various events like participant joins/leaves, chat, etc.
125+
// This is optional
126+
meeting.rtkMeeting.participants.joined.on('participantJoined', (participant) => {
127+
textProcessor.speak(`Participant Joined ${participant.name}`);
128+
});
129+
meeting.rtkMeeting.participants.joined.on('participantLeft', (participant) => {
130+
textProcessor.speak(`Participant Left ${participant.name}`);
131+
});
132+
133+
// Make sure to actually join the meeting after registering all handlers
134+
await meeting.rtkMeeting.join();
135+
}
136+
137+
async deinit() {
138+
// Add any other cleanup logic required
139+
await this.deinitPipeline();
140+
}
141+
}
142+
143+
export default {
144+
async fetch(request, env, _ctx): Promise<Response> {
145+
const url = new URL(request.url);
146+
const meetingId = url.searchParams.get('meetingId');
147+
if (!meetingId) {
148+
return new Response(null, { status: 400 });
149+
}
150+
151+
const stub = env.ROOM_OBJECT.get(env.ROOM_OBJECT.idFromName(meetingId));
152+
// The fetch method is implemented for handling internal pipeline logic
153+
if (url.pathname.startsWith('/agentsInternal')) {
154+
return stub.fetch(request);
155+
}
156+
157+
// Your logic continues here
158+
switch (url.pathname) {
159+
case '/init':
160+
// This is the authToken for joining a meeting, it can be passed
161+
// in query parameters as well if needed
162+
const authHeader = request.headers.get('Authorization');
163+
if (!authHeader) {
164+
return new Response(null, { status: 401 });
165+
}
166+
167+
// We just need the part after `Bearer `
168+
await stub.init(meetingId, authHeader.split(' ')[1], url.host);
169+
170+
return new Response(null, { status: 200 });
171+
case '/deinit':
172+
await stub.deinit();
173+
return new Response(null, { status: 200 });
174+
}
175+
176+
return new Response(null, { status: 404 });
177+
},
178+
} satisfies ExportedHandler<Env>;
179+
```
180+
</TypeScriptExample>
181+
182+
The Realtime Agents SDK provides several elements that work together to create an end-to-end pipeline
183+
184+
- `Meeting`: Represents a RealtimeKit meeting that will be joined by the agent
185+
186+
- `DeepgramSTT`: Takes in meeting audio and provides transcripts powered by Deepgram
187+
188+
- `TextProcessor`: A concrete implementation for this element needs to be provided by the user as it is responsible for processing the text generated in the meeting and sending back responses. We have implemented it in the `MyTextProcessor` class
189+
190+
- `ElevenLabsTTS`: Converts the generated responses to audio to be spoken in the meeting
191+
192+
We use all of these elements together to create a simple chatbot-like pipeline. As a pre-requisite, we require the meeting ID to be joined along with an authorization token for joining the meeting, which is passed during the worker invocation
193+
194+
Additionally, our class must extend `ArtificialObject` as it contains certain internal logic to handle interactions with our pipeline backend
195+
196+
## 5. Deploy your AI Worker
197+
198+
Before deploying your AI Worker globally, log in with your Cloudflare account by running:
199+
200+
```sh
201+
npx wrangler login
202+
```
203+
204+
You will be directed to a web page asking you to log in to the Cloudflare dashboard. After you have logged in, you will be asked if Wrangler can make changes to your Cloudflare account. Scroll down and select **Allow** to continue.
205+
206+
Finally, deploy your Worker to make your project accessible on the Internet. To deploy your Worker, run:
207+
208+
```sh
209+
npx wrangler deploy
210+
```
211+
212+
```sh output
213+
https://hello-agent.<YOUR_SUBDOMAIN>.workers.dev
214+
```
215+
216+
Finally, invoke the worker to make the agent join a meeting:
217+
218+
```sh
219+
curl -X POST https://hello-agent.<YOUR_SUBDOMAIN>.workers.dev/init?meetingId=<REALTIME_KIT_MEETING_ID> -H "Authorization: Bearer <REALTIME_KIT_AUTH_TOKEN>"
220+
```
221+
222+
## Related resources
223+
224+
- [Cloudflare Developers community on Discord](https://discord.cloudflare.com) - Submit feature requests, report bugs, and share your feedback directly with the Cloudflare team by joining the Cloudflare Discord server.

0 commit comments

Comments
 (0)