You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Opening a link in your default browser: https://oauth.pipelines.cloudflare.com/oauth/login?accountId=<ACCOUNT_ID>&bucketName=[R2-BUCKET-NAME]&pipelineName=[PIPELINE-NAME]
70
+
🌀 Checking access to R2 bucket "[R2-BUCKET-NAME]"
71
+
🌀 Creating Pipeline named "[PIPELINE-NAME]"
72
+
✅ Successfully created Pipeline "[PIPELINE-NAME]" with id [PIPELINE-ID]
73
+
🎉 You can now send data to your Pipeline!
74
+
75
+
To start interacting with this Pipeline from a Worker, open your Worker’s config file and add the following binding configuration:
Copy file name to clipboardExpand all lines: src/content/docs/pipelines/index.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,7 +21,7 @@ Ingest and load real time data streams to R2, using Cloudflare Pipelines.
21
21
22
22
<Plantype="paid" />
23
23
24
-
Pipelines lets you ingest and load real time data streams into R2, without managing any infrastructure. You can send data to a Pipeline data via HTTP, or from a Worker. Your Pipeline will handle batching the data, generating compressed JSON files, and delivering the files to an R2 bucket.
24
+
Pipelines lets you ingest and load real time data streams into R2, without managing any infrastructure. You can send data to a Pipeline via HTTP, or from a Worker. Your Pipeline will handle batching the data, generating compressed JSON files, and delivering the files to an R2 bucket.
25
25
26
26
Refer to the [get started guide](/pipelines/get-started) to start building with Pipelines.
You can send records to your Pipeline directly from a [Cloudflare Worker](/workers/). To do so, you need to:
14
+
14
15
1. Create a Worker
15
16
2. Create a Pipeline
16
-
3. Add your Pipeline as a binding in your Workers' `wrangler.toml` file
17
+
3. Add your Pipeline as a binding in your Workers' `wrangler.jsonc` file
17
18
4. Write your Worker, to send records to your Pipeline
18
19
5. Deploy your Worker
19
20
6. Verify in R2
20
21
21
22
## 1. Create a Worker
23
+
22
24
Create a Cloudflare Worker if you don't already have one. This Worker will send records to your Pipeline.
23
25
24
26
To create a Worker, run the following command in a terminal:
@@ -39,38 +41,44 @@ To create a Worker, run the following command in a terminal:
39
41
}}
40
42
/>
41
43
42
-
This will create a new directory, which will include both a `src/index.ts` Worker script, and a [`wrangler.toml`](/workers/wrangler/configuration/) configuration file. Navigate into the newly created directory:
44
+
This will create a new directory, which will include both a `src/index.ts` Worker script, and a [`wrangler.jsonc`](/workers/wrangler/configuration/) configuration file. Navigate into the newly created directory:
43
45
44
46
```sh
45
47
cd pipeline-worker
46
48
```
47
49
48
50
## 2. Create a Pipeline
51
+
49
52
Create a new Pipeline, if you don't already have one. If this is your first time using Pipelines, follow the instructions in the [get started guide](/pipelines/get-started).
50
53
51
54
By default, Worker bindings are enabled on all Pipelines. Keep track of the name you gave your Pipeline in this stage; we'll use it in the next step.
52
55
53
56
## 3. Add a Binding
57
+
54
58
To connect your Worker to your Pipeline, you need to create a binding. [Bindings](/workers/runtime-apis/bindings/) allow you to grant specific capabilities to your Worker.
55
59
56
-
Open your newly generated `wrangler.toml` configuration file and add the following:
60
+
Open your newly generated `wrangler.jsonc` configuration file and add the following:
57
61
62
+
<WranglerConfig>
58
63
```toml
59
64
[[pipelines]]
60
65
binding = "MY_PIPELINE"
61
66
pipeline = "<MY-PIPELINE-NAME>"
62
67
```
68
+
</WranglerConfig>
63
69
64
70
Replace `<MY-PIPELINE-NAME>` with the name of the Pipeline you created in step 2. Next, replace `MY_PIPELINE` with the name you want for your `binding`. The binding must be a valid JavaScript variable name. This is the variable you will use to reference this queue in your Worker.
65
71
66
72
## 4. Write your Worker
73
+
67
74
You will now configure your Worker to send records to your Pipeline. Your Worker will:
68
75
69
76
1. Take a request it receives from the browser
70
77
2. Transform the request to JSON
71
78
3. Send the resulting record to your Pipeline
72
79
73
80
In your Worker project directory, open the `src` folder and add the following to your `index.ts` file:
81
+
74
82
```ts
75
83
exportinterfaceEnv {
76
84
<MY_PIPELINE>:Pipeline<any>;
@@ -94,7 +102,8 @@ Replace `MY_PIPELINE` with the name of the binding you set in Step 3. If sending
94
102
In a production application, you would likely use a [`try...catch`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/try...catch) statement to catch the exception and handle it directly (for example, return a custom error or even retry).
95
103
96
104
## 5. Publish your Worker
97
-
With your `wrangler.toml` file and `index.ts` file configured, you are ready to publish your Worker. To publish your Worker, run:
105
+
106
+
With your `wrangler.jsonc` file and `index.ts` file configured, you are ready to publish your Worker. To publish your Worker, run:
98
107
99
108
```sh
100
109
npx wrangler deploy
@@ -111,9 +120,11 @@ Published <YOUR-WORKER-NAME> (0.29 sec)
111
120
Copy your `*.workers.dev` subdomain and paste it into a new browser tab. Refresh the page a few times to send records to your Pipeline. Your browser should return the `Success` response after sending the record to your Pipeline.
112
121
113
122
## 6. Verify in R2
123
+
114
124
Go to the R2 bucket you created in step 2 via [the Cloudflare dashboard](https://dash.cloudflare.com/). You should see a prefix for today's date. Click through, and you'll find one or more files, containing the records you sent in step 4.
115
125
116
126
# Local Development
127
+
117
128
:::note
118
-
Known issue: When running your Worker locally, sending data to your Pipeline currently results in an error.
129
+
When running your Worker locally, the data is currently not sent to your production Pipeline. Only when you deploy your Worker will the data be sent to your production Pipeline.
In this tutorial, you will learn how to ingest clickstream data to a R2 bucket using Pipelines. You will also learn how to connect the bucket to MotherDuck. You will then query the data using MotherDuck.
18
+
In this tutorial, you will learn how to ingest clickstream data to a R2 bucket using Pipelines. You will use the Pipeline binding to send the clickstream data to the R2 bucket from your Worker. You will also learn how to connect the bucket to MotherDuck. You will then query the data using MotherDuck.
19
19
20
20
For this tutorial, you will build a landing page of an e-commerce website. The page will list the products available for sale. A user can click on the view button to view the product details or click on the add to cart button to add the product to their cart. The focus of this tutorial is to show how to ingest the data to R2 and query it using MotherDuck. Hence, the landing page will be a simple HTML page with no functionality.
21
21
@@ -49,7 +49,7 @@ Create a new Worker project by running the following commands:
49
49
product="workers"
50
50
params={{
51
51
category: "hello-world",
52
-
type: "Hello World Worker",
52
+
type: "Worker + Assets",
53
53
lang: "TypeScript",
54
54
}}
55
55
/>
@@ -60,16 +60,9 @@ Navigate to the `e-commerce-pipelines` directory:
60
60
cd e-commerce-pipelines
61
61
```
62
62
63
-
## 2. Create the front-end
63
+
## 2. Update the front-end
64
64
65
-
Using Static Assets, you can serve the frontend of your application from your Worker. To use Static Assets, you need to add the required bindings to your `wrangler.toml` file.
66
-
67
-
```toml
68
-
[assets]
69
-
directory = "public"
70
-
```
71
-
72
-
Next, create a `public` directory and add an `index.html` file. The `index.html` file should contain the following HTML code:
65
+
Using Static Assets, you can serve the frontend of your application from your Worker. The above step creates a new Worker project with a default `public/index.html` file. Update the `public/index.html` file with the following HTML code:
73
66
74
67
<details>
75
68
<summary>Select to view the HTML code</summary>
@@ -309,19 +302,35 @@ You need to create a new pipeline and connect it to your R2 bucket.
309
302
Create a new pipeline `clickstream-pipeline` using the [Wrangler CLI](/workers/wrangler/):
Replace `<BUCKET_NAME>` with the name of your R2 bucket.
316
309
317
-
When you run the command, you will be prompted to authorize Cloudflare Workers Pipelines to create R2 API tokens on your behalf. These tokens are required by your Pipeline. Your Pipeline uses these tokens when loading data into your bucket. You can approve the request through the browser link which will open automatically.
310
+
When you run the command, you will be prompted to authorize Pipelines to create R2 API tokens on your behalf. These tokens are required by your Pipeline. Your Pipeline uses these tokens when loading data into your bucket. You can approve the request through the browser link which will open automatically.
318
311
319
312
```output
320
-
🌀 Authorizing R2 bucket <BUCKET_NAME>
321
-
🌀 Creating pipeline named "clickstream-pipeline"
322
-
✅ Successfully created pipeline "clickstream-pipeline" with id <PIPELINES_ID>
Opening a link in your default browser: https://oauth.pipelines.cloudflare.com/oauth/login?accountId=<ACCOUNT_ID>&bucketName=<BUCKET_NAME>&pipelineName=clickstream-pipeline
315
+
🌀 Checking access to R2 bucket "<BUCKET_NAME>"
316
+
🌀 Creating Pipeline named "clickstream-pipeline"
317
+
✅ Successfully created Pipeline "clickstream-pipeline" with id <PIPELINE_ID>
318
+
🎉 You can now send data to your Pipeline!
319
+
320
+
To start interacting with this Pipeline from a Worker, open your Worker’s config file and add the following binding configuration:
@@ -336,7 +345,7 @@ import { WranglerConfig } from "~/components";
336
345
337
346
```toml
338
347
[[pipelines]]
339
-
binding = "MY_PIPELINE"
348
+
binding = "PIPELINE"
340
349
pipeline = "clickstream-pipeline"
341
350
```
342
351
@@ -346,7 +355,7 @@ Next, update the type in the `worker-configuration.d.ts` file. Add the following
346
355
347
356
```ts title="worker-configuration.d.ts"
348
357
interfaceEnv {
349
-
MY_PIPELINE:Pipeline;
358
+
PIPELINE:Pipeline;
350
359
}
351
360
```
352
361
@@ -360,7 +369,7 @@ export default {
360
369
if (pathname==="/api/clickstream"&&method==="POST") {
361
370
const body = (awaitrequest.json()) as { data:any };
362
371
try {
363
-
awaitenv.MY_PIPELINE.send([body.data]);
372
+
awaitenv.PIPELINE.send([body.data]);
364
373
returnnewResponse("OK", { status: 200 });
365
374
} catch (error) {
366
375
console.error(error);
@@ -466,4 +475,4 @@ This project serves as a foundation for building scalable, data-driven applicati
466
475
467
476
For your next steps, consider exploring more advanced querying techniques with MotherDuck, implementing real-time analytics, or integrating additional Cloudflare services to further optimize your application's performance and security.
468
477
469
-
You can find the source code of the application in the [GitHub repository](https://github.com/harshil1712/e-commerce-clickstream).
478
+
You can find the source code of the application in the [GitHub repository](https://github.com/harshil1712/cf-pipelines-bindings-demo).
Copy file name to clipboardExpand all lines: src/content/docs/pipelines/tutorials/send-data-from-client/index.mdx
+4-15Lines changed: 4 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,5 @@
1
1
---
2
-
updated: 2025-03-03
2
+
updated: 2025-03-31
3
3
difficulty: Intermediate
4
4
content_type: 📝 Tutorial
5
5
pcx_content_type: tutorial
@@ -48,7 +48,7 @@ Create a new Worker project by running the following commands:
48
48
product="workers"
49
49
params={{
50
50
category: "hello-world",
51
-
type: "Hello World Worker",
51
+
type: "Worker + Assets",
52
52
lang: "TypeScript",
53
53
}}
54
54
/>
@@ -59,20 +59,9 @@ Navigate to the `e-commerce-pipelines-client-side` directory:
59
59
cd e-commerce-pipelines-client-side
60
60
```
61
61
62
-
## 2. Create the front-end
62
+
## 2. Update the front-end
63
63
64
-
Using Static Assets, you can serve the frontend of your application from your Worker. To use Static Assets, you need to add the required bindings to your `wrangler.toml` file.
65
-
66
-
<WranglerConfig>
67
-
68
-
```toml
69
-
[assets]
70
-
directory = "public"
71
-
```
72
-
73
-
</WranglerConfig>
74
-
75
-
Next, create a `public` directory and add an `index.html` file. The `index.html` file should contain the following HTML code:
64
+
Using Static Assets, you can serve the frontend of your application from your Worker. The above step creates a new Worker project with a default `public/index.html` file. Update the `public/index.html` file with the following HTML code:
0 commit comments