|
8 | 8 | content: Get started |
9 | 9 | --- |
10 | 10 |
|
11 | | -Test |
| 11 | +import { Render, PackageManagers } from "~/components"; |
| 12 | + |
| 13 | +:::note |
| 14 | + |
| 15 | +Pipelines is in **public beta**, and any developer with a [paid Workers plan](/workers/platform/pricing/#workers) can start using Pipelines immediately. |
| 16 | + |
| 17 | +Pipelines let you ingest real time streaming data, and store it in R2. By following this guide, you'll create your first Pipeline, connect it to your R2 bucket, and post data to it via HTTP. |
| 18 | + |
| 19 | +::: |
| 20 | + |
| 21 | +## Prerequisites |
| 22 | + |
| 23 | +To use Pipelines, you will need: |
| 24 | + |
| 25 | +<Render file="prereqs" product="workers" /> |
| 26 | + |
| 27 | +### 1. Enable Pipelines |
| 28 | + |
| 29 | +TODO |
| 30 | + |
| 31 | +### 2. Install Wrangler |
| 32 | +TODO |
| 33 | + |
| 34 | +## 1. Setup an R2 bucket to use as a destination |
| 35 | +Pipelines is built to ingest data, and store it in an R2 bucket. Create a bucket, following our [Get Started Guide for R2](https://developers.cloudflare.com/r2/get-started/) if you need to. Save the bucket name; you'll need it for the next sep. |
| 36 | + |
| 37 | +## 2. Create a Pipeline |
| 38 | +To create a pipeline using Wrangler, run this command in a shell. Specify the name of your pipeline, as well as the name of the R2 bucket you created in Step 1. |
| 39 | + |
| 40 | +```sh |
| 41 | +npx wrangler pipelines create <PIPELINE-NAME> --r2 <R2-BUCKET-NAME> |
| 42 | +``` |
| 43 | + |
| 44 | +Choose a descrpitive name for your Pipeline, related to the type of events you intend to ingest. You cannot change the Pipeline name after you have set it. |
| 45 | + |
| 46 | +TODO: Pipeline name limitations |
| 47 | + |
| 48 | +Once your pipeline is created, you'll receive an HTTP endpoint which you can post data to. You should see output which resembles the below: |
| 49 | + |
| 50 | +```sh |
| 51 | +🌀 Authorizing R2 bucket "<R2-BUCKET-NAME>" |
| 52 | +🌀 Creating pipeline named "<PIPELINE-NAME>" |
| 53 | +✅ Successfully created pipeline <PIPELINE-NAME> with ID <PIPELINE-ID> |
| 54 | + |
| 55 | +You can now send data to your pipeline with: |
| 56 | + curl "https://<PIPELINE-ID>.pipelines.cloudflare.com/" -d '[{ ...JSON_DATA... }]' |
| 57 | +``` |
| 58 | + |
| 59 | +## 3. Post data to your pipeline |
| 60 | + |
| 61 | +Use a curl command in your terminal to post an array of JSON objects to the endpoint you received in Step 1. |
| 62 | + |
| 63 | +```sh |
| 64 | +curl -H "Content-Type:application/json" \ |
| 65 | + -d '[{"account_id":"test", "other_data": "test"},{"account_id":"test","other_data": "test2"}]' \ |
| 66 | + <HTTP-endpoint> |
| 67 | +``` |
| 68 | + |
| 69 | +Once the data has been successfully accepted by the Pipeline, you'll receive a success message. |
| 70 | + |
| 71 | +Pipelines handles batching the data, so you can continue posting data to the Pipeline. Once a batch is filled up, the data will be partitioned by date, and written to your R2 bucket. |
| 72 | + |
| 73 | +## 4. Verify in R2 |
| 74 | +Navigate to the dashboard for the R2 bucket that you created in step 1. You should see a prefix for today's date. Click through, and you'll see a file created containing the JSON data you sent in Step 3. |
| 75 | + |
| 76 | +By completing this guide, you've got a pipeline, with an HTTP endpoint as a source, and an R2 bucket as a destination. |
0 commit comments