Skip to content

Commit 09646a4

Browse files
committed
Improvements to overview
1 parent fe1aefa commit 09646a4

File tree

2 files changed

+13
-16
lines changed

2 files changed

+13
-16
lines changed

src/content/docs/pipelines/concepts/how-pipelines-work.mdx

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -29,15 +29,8 @@ Ingested data is buffered until a sufficiently large batch of data has accumulat
2929

3030
Finally, the the batch of data is converted into output files, which are compressed, and delivered to the configured R2 bucket. Any transient failures, such as network failures, are automatically retried.
3131

32-
Once files have been successfully written to R2, the buffers are flushed.
33-
3432
## How a Pipeline handles updates
35-
Data delivery is guaranteed even while updating an existing pipeline. Updating an existing pipeline effectively creates a new deployment, including all your previously configured options. Requests are gracefully re-routed to the new pipeline. The old pipeline continues to write data into your destination. We spin down the old pipeline only after all the data has been written out. This way, you won’t lose data even while updating a pipeline.
36-
37-
## How Pipelines scale
38-
Pipelines are organized into shards. You can [customize the number of shards](/pipelines/build-with-pipelines/shards) to increase maximum throughput, or to reduce the number of output files generated.
39-
40-
Each shard consists of layers of durable objects. Shards are stateless, so your pipeline can scale by horizontally scaling the number of shards.
33+
Data delivery is guaranteed even while updating an existing pipeline. Updating an existing pipeline effectively creates a new deployment, including all your previously configured options. Requests are gracefully re-routed to the new pipeline. The old pipeline continues to write data into your destination. Once the old pipeline is fully drained, it is spun down.
4134

4235
## What if I send too much data? Do Pipelines communicate backpressure?
4336
If you send too much data, the pipeline will communicate backpressure by returning a 429 response to HTTP requests, or throwing an error if using the Workers API. Refer to the [limits](/pipelines/platform/limits) to learn how much volume a single pipeline can support.

src/content/docs/pipelines/index.mdx

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,10 @@ Ingest real time data streams, and load to R2, using Cloudflare Pipelines.
2121

2222
<Plan type="paid" />
2323

24-
Cloudflare Pipelines lets you ingest high volumes of real time data, without managing any infrastructure. A single Pipeline can ingest up to 100 MB of data per second. Ingested data is automatically batched, written to output files, and delivered to an R2 bucket in your account.
24+
Cloudflare Pipelines lets you ingest high volumes of real time data, without managing any infrastructure. A single pipeline can ingest up to 100 MB of data per second. Ingested data is automatically batched, written to output files, and delivered to an R2 bucket in your account. You can use Pipelines to build a data lake of clickstream data, or to archive logs from a service.
2525

26-
## Build your first Pipeline
27-
You can create a Pipeline with a single command.
26+
## Create your first pipeline
27+
You can setup a pipeline to ingest data via HTTP, and deliver output to R2, with a single command:
2828

2929
```sh
3030
npx wrangler pipelines create my-pipeline --r2-bucket my-r2-bucket
@@ -37,17 +37,21 @@ You can now send data to your pipeline with:
3737
curl "https://0adedf710c20401x74380236286a8b9c.pipelines.cloudflare.com/" -d '[{ "foo":"bar }]'
3838
```
3939

40-
Refer to the [getting started guide](/pipelines/getting-started) to start building with Pipelines.
40+
Refer to the [getting started guide](/pipelines/getting-started) to start building with pipelines.
41+
42+
:::note
43+
While in beta, you will not be billed for Pipelines usage. You will be billed only for [R2 operations](r2/pricing/).
44+
:::
4145

4246
***
4347
## Features
4448

4549
<Feature header="HTTP as a source" href="/pipelines/build-with-pipelines/http/">
46-
Each Pipeline generates a globally scalable HTTP endpoint, which supports authentication and CORS settings.
50+
Each pipeline generates a globally scalable HTTP endpoint, which supports authentication and CORS settings.
4751
</Feature>
4852

49-
<Feature header="Customize batch behavior, deliver to a prefix, and receive partitioned data" href="/pipelines/build-with-pipelines/output-settings">
50-
Customize Pipeline settings to generate output files that are efficient to query.
53+
<Feature header="Customize output settings" href="/pipelines/build-with-pipelines/output-settings">
54+
Define batch sizes and enable compression to generate output files that are efficient to query.
5155
</Feature>
5256

5357
***
@@ -73,7 +77,7 @@ Cloudflare Workers allows developers to build serverless applications and deploy
7377
<CardGrid>
7478

7579
<LinkTitleCard title="Limits" href="/pipelines/platform/limits/" icon="document">
76-
Learn about Pipelines limits.
80+
Learn about pipelines limits.
7781
</LinkTitleCard>
7882

7983
<LinkTitleCard title="@CloudflareDev" href="https://x.com/cloudflaredev" icon="x.com">

0 commit comments

Comments
 (0)