Skip to content

Commit 047106b

Browse files
committed
Updated metrics documentation
1 parent 38a5bc1 commit 047106b

File tree

2 files changed

+100
-40
lines changed

2 files changed

+100
-40
lines changed

src/content/docs/pipelines/observability/metrics.mdx

Lines changed: 40 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,15 +6,48 @@ sidebar:
66

77
---
88

9-
Pipelines metrics are split across three different nodes under `viewer` > `accounts`. Refer to [Explore the GraphQL schema](/analytics/graphql-api/getting-started/explore-graphql-schema/) to learn how to navigate a GraphQL schema and discover which data are available.
9+
Pipelines expose metrics which allow you to measure data ingested, requests made, and data delivered, across all your Pipelines.
1010

11-
To learn more about the GraphQL Analytics API, refer to [GraphQL Analytics API](/analytics/graphql-api/).
11+
The metrics displayed in the [Cloudflare dashboard](https://dash.cloudflare.com/) are queried from Cloudflare’s [GraphQL Analytics API](/analytics/graphql-api/). You can access the metrics [programmatically](#query-via-the-graphql-api) via GraphQL or HTTP client.
1212

13-
You can use the GraphQL API to measure metrics for data ingested, as well as data delivered.
13+
## Metrics
14+
### Ingestion
15+
Pipelines export the below metrics within the `pipelinesIngestionAdaptiveGroups` dataset.
1416

15-
## Write GraphQL queries
17+
| Metric | GraphQL Field Name | Description |
18+
| ---------------------- | ------------------------- | ---------------------------------------------------------------|
19+
| Ingestion Events | `count` | Number of ingestion events, or requests made, to a pipeline. |
20+
| Ingested Bytes | `ingestedBytes` | Total number of bytes ingested |
21+
| Ingested Records | `ingestedRecords` | Total number of records ingested |
1622

17-
Examples of how to explore your Pipelines metrics.
23+
The `pipelinesIngestionAdaptiveGroups` dataset provides the following dimensions for filtering and grouping queries:
24+
* `pipelineId` - ID of the pipeline
25+
* `datetime` - Timestamp of the ingestion event
26+
* `date` - Timestamp of the ingestion event, truncated to the start of a day
27+
* `datetimeHour` - Timestamp of the ingestion event, truncated to the start of an hour
28+
* `datetimeMinute` - Timestamp of the ingestion event, truncated to the start of a minute
29+
30+
### Delivery
31+
Pipelines export the below metrics within the `pipelinesDeliveryAdaptiveGroups` dataset.
32+
33+
| Metric | GraphQL Field Name | Description |
34+
| ---------------------- | ------------------------- | ---------------------------------------------------------------|
35+
| Ingestion Events | `count` | Number of delivery events to an R2 bucket |
36+
| Delivered Bytes | `deliveredBytes` | Total number of bytes ingested |
37+
38+
The `pipelinesDeliverynAdaptiveGroups` dataset provides the following dimensions for filtering and grouping queries:
39+
* `pipelineId` - ID of the pipeline
40+
* `datetime` - Timestamp of the delivery event
41+
* `date` - Timestamp of the delivery event, truncated to the start of a day
42+
* `datetimeHour` - Timestamp of the delivery event, truncated to the start of an hour
43+
* `datetimeMinute` - Timestamp of the delivery event, truncated to the start of a minute
44+
45+
46+
## Query via the GraphQL API
47+
48+
You can programmatically query analytics for your Workflows via the [GraphQL Analytics API](/analytics/graphql-api/). This API queries the same datasets as the Cloudflare dashboard, and supports GraphQL [introspection](/analytics/graphql-api/features/discovery/introspection/).
49+
50+
Pipelines GraphQL datasets require an `accountTag` filter, with your Cloudflare account ID.
1851

1952
### Measure total bytes & records ingested over time period
2053

@@ -45,13 +78,13 @@ query PipelineIngestion($accountTag: string!, $pipelineId: string!, $datetimeSta
4578
### Measure volume of data delivered
4679

4780
```graphql
48-
query PipelineDelivery($accountTag: string!, $queueId: string!, $datetimeStart: Time!, $datetimeEnd: Time!) {
81+
query PipelineDelivery($accountTag: string!, $pipelineId: string!, $datetimeStart: Time!, $datetimeEnd: Time!) {
4982
viewer {
5083
accounts(filter: {accountTag: $accountTag}) {
5184
pipelinesDeliveryAdaptiveGroups(
5285
limit: 10000
5386
filter: {
54-
pipelineId: $queueId
87+
pipelineId: $pipelineId
5588
datetime_geq: $datetimeStart
5689
datetime_leq: $datetimeEnd
5790
}

src/content/docs/workers/wrangler/commands.mdx

Lines changed: 60 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -2019,7 +2019,7 @@ Pipelines is currently in open beta. Report Pipelines bugs in [GitHub](https://g
20192019

20202020
:::
20212021

2022-
Manage your [Pipelines](/pipelines/) configurations.
2022+
Manage your [Pipelines](/pipelines/).
20232023

20242024
### `create`
20252025

@@ -2029,24 +2029,32 @@ Create a new pipeline
20292029
wrangler pipelines create <name> --r2 <r2-bucket-name> [OPTIONS]
20302030
```
20312031

2032-
- `name` string required
2032+
- `name` <Type text="string" /> <MetaInfo text="required" />
20332033
- The name of the pipeline to create
2034-
- `--r2` string required
2034+
- `--source` <Type text="array" /> <MetaInfo text="optional" />
2035+
- List of alowed sources. Options: 'http' or 'worker'
2036+
- `--require-http-auth` <Type text="boolean" /> <MetaInfo text="optional" />
2037+
- Require Cloudflare API token to authenticate with the HTTPS endpoint. Defaults to false.
2038+
- `--cors-origins` <Type text="array" /> <MetaInfo text="optional" />
2039+
- CORS Origin allowlist for HTTP endpoint. Allows `*`. Defaults to an empty array.
2040+
- `--batch-max-mb` <Type text="number" /> <MetaInfo text="optional" />
2041+
- The maximum size a batch before data is written, in megabytes. Defaults to 100. Must be between 0.001 and 100.
2042+
- `--batch-max-rows` <Type text="number" /> <MetaInfo text="optional" />
2043+
- The maximum number of rows in a batch before data is written. Defaults to 10000000. Must be between 1 and 10000000.
2044+
- `--batch-max-seconds` <Type text="number" /> <MetaInfo text="optional" />
2045+
- The maximum duration of a batch before data is written, in seconds. Defaults to 300. Must be between 1 and 300.
2046+
- `--r2-bucket` <Type text="string" /> <MetaInfo text="required" />
20352047
- The name of the R2 bucket used as the destination to store the data.
2036-
- `--batch-max-mb` number optional
2037-
- The maximum size a batch before data is written, in megabytes. Default 10 MB, maximum 100 MB.
2038-
- `--batch-max-rows` number optional
2039-
- The maximum number of rows in a batch before data is written. Default, and maximum, 10,000.
2040-
- `--batch-max-seconds` number optional
2041-
- The maximum duration of a batch before data is written, in seconds. Default 15s. Maximum of 600s.
2042-
- `--compression` string optional
2048+
- `--r2-bucket-access-key-id` <Type text="string" /> <MetaInfo text="optional" />
2049+
- Access key ID used to authenticate with R2. Leave empty for oauth confirmation.
2050+
- `--r2-bucket-secret-access-key` <Type text="string" /> <MetaInfo text="optional" />
2051+
- Secret access key ID used to authenticate with R2. Leave empty for oauth confirmation.
2052+
- `--r2-prefix` <Type text="string" /> <MetaInfo text="optional" />
2053+
- Prefix for storing files in the destination bucket.
2054+
- `--compression` <Type text="string" /> <MetaInfo text="optional" />
20432055
- Type of compression to apply to output files. Choices: "none", "gzip", "deflate"
2044-
- `--prefix` string optional
2045-
- Optional base path to store files in the destination bucket.
2046-
- `--filepath` string optional
2047-
- The path to store partitioned files in the destination bucket. Defaults to `event_date=${date}/hr=${hr}`
2048-
- `--filename` string optional
2049-
- The name of the file in the bucket. Must contain `${slug}`. File extension is optional. Defaults to `${slug}${extension}`
2056+
- `--shard-count` <Type text="number" /> <MetaInfo text="optional" />
2057+
- Number of pipeline shards. More shards handle higher request volume; fewer shards produce larger output files. Defaults to 2. Must be between 1 and 15.
20502058

20512059
### `update`
20522060

@@ -2056,24 +2064,43 @@ Update an existing pipeline
20562064
wrangler pipelines update <name> [OPTIONS]
20572065
```
20582066

2059-
- `name` string required
2060-
- The name of the pipeline to update
2061-
- `--r2` string required
2067+
- `name` <Type text="string" /> <MetaInfo text="required" />
2068+
- The name of the pipeline to create
2069+
- `--source` <Type text="array" /> <MetaInfo text="optional" />
2070+
- List of alowed sources. Options: 'http' or 'worker'
2071+
- `--require-http-auth` <Type text="boolean" /> <MetaInfo text="optional" />
2072+
- Require Cloudflare API token to authenticate with the HTTPS endpoint. Defaults to false.
2073+
- `--cors-origins` <Type text="array" /> <MetaInfo text="optional" />
2074+
- CORS Origin allowlist for HTTP endpoint. Allows `*`. Defaults to an empty array.
2075+
- `--batch-max-mb` <Type text="number" /> <MetaInfo text="optional" />
2076+
- The maximum size a batch before data is written, in megabytes. Defaults to 100. Must be between 0.001 and 100.
2077+
- `--batch-max-rows` <Type text="number" /> <MetaInfo text="optional" />
2078+
- The maximum number of rows in a batch before data is written. Defaults to 10000000. Must be between 1 and 10000000.
2079+
- `--batch-max-seconds` <Type text="number" /> <MetaInfo text="optional" />
2080+
- The maximum duration of a batch before data is written, in seconds. Defaults to 300. Must be between 1 and 300.
2081+
- `--r2-bucket` <Type text="string" /> <MetaInfo text="required" />
20622082
- The name of the R2 bucket used as the destination to store the data.
2063-
- `--batch-max-mb` number optional
2064-
- The maximum size a batch before data is written, in megabytes. Default 10 MB, maximum 100 MB.
2065-
- `--batch-max-rows` number optional
2066-
- The maximum number of rows in a batch before data is written. Default, and maximum, 10,000.
2067-
- `--batch-max-seconds` number optional
2068-
- The maximum duration of a batch before data is written, in seconds. Default 15s. Maximum of 600s.
2069-
- `--compression` string optional
2083+
- `--r2-bucket-access-key-id` <Type text="string" /> <MetaInfo text="optional" />
2084+
- Access key ID used to authenticate with R2. Leave empty for oauth confirmation.
2085+
- `--r2-bucket-secret-access-key` <Type text="string" /> <MetaInfo text="optional" />
2086+
- Secret access key ID used to authenticate with R2. Leave empty for oauth confirmation.
2087+
- `--r2-prefix` <Type text="string" /> <MetaInfo text="optional" />
2088+
- Prefix for storing files in the destination bucket.
2089+
- `--compression` <Type text="string" /> <MetaInfo text="optional" />
20702090
- Type of compression to apply to output files. Choices: "none", "gzip", "deflate"
2071-
- `--prefix` string optional
2072-
- Optional base path to store files in the destination bucket.
2073-
- `--filepath` string optional
2074-
- The path to store partitioned files in the destination bucket. Defaults to `event_date=${date}/hr=${hr}`
2075-
- `--filename` string optional
2076-
- The name of the file in the bucket. Must contain `${slug}`. File extension is optional. Defaults to `${slug}${extension}`
2091+
- `--shard-count` <Type text="number" /> <MetaInfo text="optional" />
2092+
- Number of pipeline shards. More shards handle higher request volume; fewer shards produce larger output files. Defaults to 2. Must be between 1 and 15.
2093+
2094+
### `get`
2095+
2096+
Get the configuration for an existing pipeline.
2097+
2098+
```txt
2099+
wrangler pipelines get <name> [OPTIONS]
2100+
```
2101+
2102+
- `name` <Type text="string" /> <MetaInfo text="required" />
2103+
- The name of the pipeline to inspect
20772104

20782105
### `delete`
20792106

@@ -2083,7 +2110,7 @@ Deletes an existing pipeline
20832110
wrangler pipelines delete <name> [OPTIONS]
20842111
```
20852112

2086-
- `name` string required
2113+
- `name` <Type text="string" /> <MetaInfo text="required" />
20872114
- The name of the pipeline to delete
20882115

20892116
### `list`

0 commit comments

Comments
 (0)