WIP: Pipelines Docs

src/content/changelogs/pipelines.yaml

@@ -0,0 +1,11 @@
+---
+link: "/pipelines/reference/changelog/"
+productName: Pipelines
+productLink: "/pipelines/"
+productArea: Developer Platform
+productAreaLink: "/pipelines/"
+entries:
+  - publish_date: "2024-09-24"
+    title: Pipelines is now in public beta.
+    description: |-
+      Pipelines, a new product to ingest and store real time streaming data, is now in public beta. The public beta is avaiable to any user with a [free or paid Workers plan](/workers/platform/pricing/). Create a Pipeline, and you'll be able to post data to it via HTTP or from a Cloudflare Worker. Pipelines handles batching, buffering and partitioning the data, before writing it to an R2 bucket of your choice. It's useful to collect clickstream data, or ingest logs from a service. Start building with our [get started guide](/pipelines/getting-started/).

src/content/docs/pipelines/examples/index.mdx

@@ -0,0 +1,11 @@
+---
+title: Examples
+pcx_content_type: navigation
+sidebar:
+  order: 4
+
+---
+
+import { DirectoryListing } from "~/components"
+
+<DirectoryListing />

src/content/docs/pipelines/get-started.mdx

@@ -0,0 +1,78 @@
+---
+title: Get started
+pcx_content_type: get-started
+sidebar:
+  order: 2
+head:
+  - tag: title
+    content: Get started
+---
+
+import { Render, PackageManagers } from "~/components";
+
+:::note
+
+Pipelines is in **public beta**, and any developer with a [paid Workers plan](/workers/platform/pricing/#workers) can start using Pipelines immediately.
+
+Pipelines let you ingest real time streaming data, and store it in R2. Pipeline handles batching, partitioning, and optionally compressing your final files By following this guide, you'll create your first Pipeline, connect it to your R2 bucket, and post data to it via HTTP.
+
+:::
+
+## Prerequisites
+
+To use Pipelines, you will need:
+
+<Render file="prereqs" product="workers" />
+
+### 1. Enable Pipelines
+
+TODO
+
+### 2. Install Wrangler
+TODO
+
+## 1. Setup an R2 bucket to use as a destination
+Pipelines is built to ingest data, and store it in an R2 bucket. Create a bucket, following our [Get Started Guide for R2](r2/get-started/) if you need to. Save the bucket name; you'll need it for the next sep.
+
+## 2. Create a Pipeline
+To create a pipeline using Wrangler, run this command in a shell. Specify the name of your pipeline, as well as the name of the R2 bucket you created in Step 1.
+
+```sh
+npx wrangler pipelines create <PIPELINE-NAME> --r2 <R2-BUCKET-NAME>
+```
+
+Choose a descrpitive name for your Pipeline, related to the type of events you intend to ingest. You cannot change the Pipeline name after you have set it.
+
+Pipeline names must be 1 to 63 characters long. Queue names cannot contain special characters outside dashes (`-`), and must start and end with a letter or number.
+
+TODO: Pipeline name limitations
+
+Once your pipeline is created, you'll receive an HTTP endpoint which you can post data to. You should see output which resembles the below:
+
+```sh
+🌀 Authorizing R2 bucket "<R2-BUCKET-NAME>"
+🌀 Creating pipeline named "<PIPELINE-NAME>"
+✅ Successfully created pipeline <PIPELINE-NAME> with ID <PIPELINE-ID>
+
+You can now send data to your pipeline with:
+  curl "https://<PIPELINE-ID>.pipelines.cloudflare.com/" -d '[{ ...JSON_DATA... }]'
+```
+
+## 3. Post data to your pipeline
+
+Use a curl command in your terminal to post an array of JSON objects to the endpoint you received in Step 1.
+
+```sh
+curl -H "Content-Type:application/json" \
+    -d '[{"account_id":"test", "other_data": "test"},{"account_id":"test","other_data": "test2"}]' \
+    <HTTP-endpoint>
+```
+
+Once the data has been successfully accepted by the Pipeline, you'll receive a success message.
+
+Pipelines handles batching the data, so you can continue posting data to the Pipeline. Once a batch is filled up, the data will be partitioned by date, and written to your R2 bucket.
+
+## 4. Verify in R2
+Navigate to the dashboard for the R2 bucket that you created in step 1. You should see a prefix for today's date. Click through, and you'll see a file created containing the JSON data you sent in Step 3.
+
+By completing this guide, you've got a pipeline, with an HTTP endpoint as a source, and an R2 bucket as a destination.

src/content/docs/pipelines/index.mdx

@@ -0,0 +1,20 @@
+---
+title: Overview
+type: overview
+pcx_content_type: overview
+sidebar:
+  order: 1
+  badge:
+    text: Beta
+head:
+  - tag: title
+    content: Pipelines
+---
+
+import { Description } from "~/components";
+
+<Description>
+
+Ingest, transform, and store real time data streams in R2.
+
+</Description>

src/content/docs/pipelines/observability/index.mdx

@@ -0,0 +1,11 @@
+---
+title: Observability
+pcx_content_type: navigation
+sidebar:
+  order: 5
+
+---
+
+import { DirectoryListing } from "~/components"
+
+<DirectoryListing />

src/content/docs/pipelines/observability/metrics.mdx

@@ -0,0 +1,9 @@
+---
+pcx_content_type: concept
+title: Metrics
+sidebar:
+  order: 10
+
+---
+
+TODO

src/content/docs/pipelines/pipelines-api.mdx

@@ -0,0 +1,7 @@
+---
+pcx_content_type: navigation
+title: Pipelines REST API
+sidebar:
+  order: 10
+
+---

src/content/docs/pipelines/reference/changelog.mdx

@@ -0,0 +1,15 @@
+---
+pcx_content_type: changelog
+title: Changelog
+changelog_file_name:
+  - pipelines
+sidebar:
+  order: 99
+
+---
+
+import { ProductChangelog } from "~/components"
+
+{/* <!-- Actual content lives in /data/changelogs/pipelines.yaml. Update the file there for new entries to appear here. For more details, refer to https://developers.cloudflare.com/style-guide/documentation-content-strategy/content-types/changelog/#yaml-file --> */}
+
+<ProductChangelog />

src/content/docs/pipelines/reference/index.mdx

@@ -0,0 +1,11 @@
+---
+pcx_content_type: navigation
+title: Platform
+sidebar:
+  order: 8
+
+---
+
+import { DirectoryListing } from "~/components"
+
+<DirectoryListing />

src/content/docs/pipelines/reference/limits.mdx

@@ -0,0 +1,8 @@
+---
+pcx_content_type: concept
+title: Limits
+sidebar:
+  order: 2
+---
+
+TODO

src/content/docs/pipelines/reference/pricing.mdx

@@ -0,0 +1,11 @@
+---
+pcx_content_type: concept
+title: Pricing
+sidebar:
+  order: 1
+head:
+	- tag: title
+		content: Pipelines Pricing
+---
+
+TODO

src/content/docs/pipelines/reference/wrangler-commands.mdx

@@ -0,0 +1,8 @@
+---
+pcx_content_type: navigation
+title: Wrangler commands
+external_link: /workers/wrangler/commands/#pipelines
+sidebar:
+  order: 80
+
+---

src/content/products/pipelines.yaml

@@ -0,0 +1,12 @@
+name: Pipelines
+
+product:
+  title: Pipelines
+  url: /pipelines/
+  group: Developer platform
+  preview_tryout: true
+
+meta:
+  title: Cloudflare Pipelines Docs
+  description: Ingest, transform, and store, real time data streams in R2.
+  author: '@cloudflare'