You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Pipelines accept arrays of valid JSON objects. You can send multiple objects in a single request, provided the total data volume is within the [documented limits](/pipelines/platform/limits). Sending data in a different format will result in an error.
49
-
50
-
For example, you can send data to your pipeline using a curl command like this:
51
-
```sh
52
-
curl -X POST https://<PIPELINE-ID>.pipelines.cloudflare.com \
You can secure your HTTP ingestion endpoint using Cloudflare API tokens. By default, authentication is turned off. To configure authentication, use the `--require-http-auth` flag while creating or updating a pipeline.
Copy file name to clipboardExpand all lines: src/content/docs/pipelines/build-with-pipelines/sources/index.mdx
+6-5Lines changed: 6 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,11 +7,13 @@ sidebar:
7
7
hideIndex: false
8
8
---
9
9
10
-
Pipelines supports the following sources:
10
+
Pipelines let you ingest data from the following sources:
11
11
*[HTTP Clients](/pipelines/build-with-pipelines/sources/http), with optional authentication and CORS settings
12
12
*[Cloudflare Workers](/workers/), using the [Pipelines Workers API](/pipelines/build-with-pipelines/sources/workers-apis)
13
13
14
-
Multiple sources can be active on a single pipeline simultaneously. For example, you can create a pipeline which accepts data from Workers and via HTTP. There is no limit to the number of source clients. Multiple workers can be configured to send data to the same pipeline.
14
+
Multiple sources can be active on a single pipeline simultaneously. For example, you can create a pipeline which accepts data from Workers and via HTTP. There is no limit to the number of source clients. Multiple Workers can be configured to send data to the same pipeline.
15
+
16
+
Each pipeline can ingest up to 100 MB/s of data, or accept up to 100,000 requests per second, across all sources.
15
17
16
18
## Configuring allowed sources
17
19
By default, ingestion via HTTP and from Workers is turned on. You can configure the allowed sources by using the `--source` flag while creating or updating a pipeline.
@@ -21,6 +23,5 @@ For example, to create a pipeline which only accepts data via a Worker, you can
HTTP ingestion URLs are tied to your pipeline ID. Turning HTTP off, and then turning it back on, will not change the URL.
26
-
:::
26
+
## Accepted data formats
27
+
Pipelines accept arrays of valid JSON objects. You can send multiple objects in a single request, provided the total data volume is within the [documented limits](/pipelines/platform/limits). Sending data in a different format will result in an error.
Pipelines exposes an API directly to your [Workers](/workers) scripts via the [bindings](/workers/runtime-apis/bindings/#what-is-a-binding) concept. Bindings allow you to securely send data to a Pipeline without having to manage API keys or clients. Sending data via a Worker is enabled by default.
14
+
Pipelines exposes an API directly to your [Workers](/workers) scripts via the [bindings](/workers/runtime-apis/bindings/#what-is-a-binding) concept. Bindings allow you to securely send data to a pipeline without having to manage API keys or clients. Sending data via a Worker is enabled by default.
pipeline = "<MY-PIPELINE-NAME>"# The name of your Pipeline
30
-
binding = "MY_PIPELINE"# The binding name, accessed using env.MY_PIPELINE
30
+
binding = "PIPELINE"# The binding name, accessed using env.MY_PIPELINE
31
31
```
32
32
33
33
</WranglerConfig>
34
34
35
35
You can bind multiple pipelines to a Worker.
36
36
37
37
### Send data
38
-
The pipelines binding exposes a `send()` method. For example, to log inbound HTTP requests to your Worker:
38
+
The Pipelines binding exposes a `send()` method. For example, to log inbound HTTP requests to your Worker:
39
39
40
40
```ts
41
41
exportdefault {
@@ -51,12 +51,9 @@ export default {
51
51
} satisfiesExportedHandler<Env>;
52
52
```
53
53
54
-
## Accepted data formats
55
-
Pipelines accept arrays of valid JSON objects. You can send multiple objects in a single request, provided the total data volume is within the [documented limits](/pipelines/platform/limits). Sending data in a different format will result in an error.
56
-
57
54
## Workers API
58
55
### `Pipeline`
59
-
A binding which allows a Worker to send messages to a Pipeline.
56
+
A binding which allows a Worker to send messages to a pipeline.
* Sends a record to the Pipeline. The body must be an array of objects which are JSON serializable.
70
-
* When the promise resolves, the message is confirmed to be stored by the pipeline.
66
+
* Sends records to the pipeline. The body must be an array of objects which are JSON serializable.
67
+
* When the promise resolves, the records are confirmed to be ingested.
71
68
72
69
:::note
73
-
When running your Worker locally, Pipelines are partially simulated. Worker code which sends data to a pipeline will execute successfully. However, the full pipeline, including batching & writing to R2, will not be executed locally.
70
+
When running your Worker locally, pipelines are partially simulated. Worker code which sends data to a pipeline will execute successfully. However, the full pipeline, including batching & writing to R2, will not be executed locally.
0 commit comments