You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Cloudflare Pipelines allows you to ingest and load high volumes of real time streaming data into [R2 Object Storage](/r2/), without managing any infrastructure. This guide will show you how to setup a Pipeline which accepts data via HTTP.
13
+
Cloudflare Pipelines allows you to ingest and load high volumes of real time streaming data into [R2 Object Storage](/r2/), without managing any infrastructure.
14
14
15
15
By following this guide, you will:
16
-
1.Create your first Pipeline.
17
-
2.Connect it to your R2 bucket.
18
-
3.Post data to it via HTTP.
19
-
4. Verify the output file written to R2.
16
+
1.Setup an R2 bucket
17
+
2.Create a pipeline, with HTTP as a source, and an R2 bucket as a sink
18
+
3.Send data to your pipeline's HTTP ingestion endpoint
19
+
4. Verify the output delivered to R2
20
20
21
21
:::note
22
22
@@ -34,50 +34,50 @@ To use Pipelines, you will need:
34
34
35
35
## 1. Set up an R2 bucket
36
36
37
-
Pipelines let you ingest records in real time, and load them into an R2 bucket. Create a bucket by following the [get started guide for R2](/r2/get-started/), or by running the command below:
37
+
Create a bucket by following the [get started guide for R2](/r2/get-started/), or by running the command below:
38
38
39
39
```sh
40
-
npx wrangler r2 bucket create [R2-BUCKET-NAME]
40
+
npx wrangler r2 bucket create clickstream-bucket
41
41
```
42
42
43
43
Save the bucket name for the next step.
44
44
45
45
## 2. Create a Pipeline
46
46
47
-
To create a Pipeline using Wrangler, run the following command in a terminal, and specify:
47
+
To create a pipeline using Wrangler, run the following command in a terminal, and specify:
After running this command, you'll be prompted to authorize Cloudflare Workers Pipelines to create R2 API tokens on your behalf. These tokens are required by your Pipeline. Your Pipeline uses the tokens when loading data into your bucket. You can approve the request through the browser link which will open automatically.
56
+
After running this command, you'll be prompted to authorize Cloudflare Workers Pipelines to create an R2 API token on your behalf. These tokens used by your pipeline when loading data into your bucket. You can approve the request through the browser link which will open automatically.
57
57
58
-
If you prefer not to authenticate this way, you may pass your [R2 API Tokens](/r2/api/s3/tokens/) to Wrangler:
58
+
If you prefer not to authenticate this way, you may pass your [R2 API Token](/r2/api/s3/tokens/) to Wrangler:
1. Ensure it is descriptive and relevant to the type of events you intend to ingest. You cannot change the name of the Pipeline after creating it.
65
+
1. Ensure it is descriptive and relevant to the type of events you intend to ingest. You cannot change the name of the pipeline after creating it.
66
66
2. Pipeline names must be between 1 and 63 characters long.
67
67
3. The name cannot contain special characters outside dashes (`-`).
68
68
4. The name must start and end with a letter or a number.
69
69
70
-
You'll notice that we have set two optional flags while creating the pipeline: `--batch-max-seconds` and `--compression`. We've added these flags to make it faster for you to see the output of your first Pipeline. For production use cases, we recommend keeping the default settings.
70
+
You'll notice that we have set two optional flags while creating the pipeline: `--batch-max-seconds` and `--compression`. We've added these flags to make it faster for you to see the output of your first pipeline. For production use cases, we recommend keeping the default settings.
71
71
72
-
Once you create your Pipeline, you will receive a HTTP endpoint which you can post data to. You should see output as shown below:
72
+
Once you create your pipeline, you will receive a HTTP endpoint which you can post data to. You should see output as shown below:
73
73
74
74
```sh
75
-
🌀 Authorizing R2 bucket "[R2-BUCKET-NAME]"
76
-
🌀 Creating pipeline named "[PIPELINE-NAME]"
77
-
✅ Successfully created pipeline [PIPELINE-NAME] with ID [PIPELINE-ID]
75
+
🌀 Authorizing R2 bucket "clickstream-bucket"
76
+
🌀 Creating pipeline named "clickstream-pipeline"
77
+
✅ Successfully created pipeline clickstream-pipeline with ID 91f312b8ca484e5db404bd3e3ef256fn
Once the Pipeline successfully accepts the data, you will receive a success message.
93
+
Once the pipeline successfully accepts the data, you will receive a success message.
94
94
95
95
Pipelines handle batching the data, so you can continue posting data to the Pipeline. Once a batch is filled up, the data will be partitioned by date, and written to your R2 bucket.
0 commit comments