Skip to content

Commit 3b1acc7

Browse files
added dash steps/tabs, moved out of r2, reformatted most of the R2 SQL docs
1 parent 579cbf2 commit 3b1acc7

File tree

16 files changed

+533
-148
lines changed

16 files changed

+533
-148
lines changed

src/content/dash-routes/index.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,7 @@
261261
},
262262
{
263263
"name": "Pipelines",
264-
"deeplink": "/?to=/:account/workers/pipelines",
264+
"deeplink": "/?to=/:account/pipelines",
265265
"parent": ["Storage & Databases"]
266266
},
267267
{

src/content/docs/r2/sql/get-started.mdx renamed to src/content/docs/r2-sql/get-started.mdx

Lines changed: 146 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ description: Learn how to get up and running with R2 SQL using R2 Data Catalog a
88
---
99
import {
1010
Render,
11+
Steps,
12+
Tabs,
13+
TabItem,
14+
DashButton,
1115
LinkCard,
1216
} from "~/components";
1317

@@ -34,43 +38,104 @@ Use a Node version manager like [Volta](https://volta.sh/) or [nvm](https://gith
3438

3539
You'll need API tokens to interact with Cloudflare services.
3640

37-
### Custom API Token
38-
1. Go to **My Profile****API Tokens** in the Cloudflare dashboard
39-
2. Select **Create Token****Custom token**
40-
3. Add the following permissions:
41-
- **Workers Pipelines** - Read, Send, Edit
42-
- **Workers R2 Storage** - Edit, Read
43-
- **Workers R2 Data Catalog** - Edit, Read
44-
- **Workers R2 SQL** - Read
41+
<Steps>
42+
1. In the Cloudflare dashboard, go to the **R2 object storage** page.
43+
<DashButton url="/?to=/:account/r2/overview" />
44+
45+
2. Select **Manage API tokens**.
46+
47+
3. Select **Create API token**.
48+
49+
4. Select the **R2 Token** text to edit your API token name.
50+
51+
5. Under **Permissions**, choose the **Admin Read & Write** permission.
52+
53+
6. Select **Create API Token**.
54+
55+
7. Note the **Token value**.
56+
57+
</Steps>
4558

4659
Export your new token as an environment variable:
4760

4861
```bash
49-
export WRANGLER_R2_SQL_AUTH_TOKEN=your_token_here
62+
export WRANGLER_R2_SQL_AUTH_TOKEN= #paste your token here
5063
```
5164

5265
If this is your first time using Wrangler, make sure to login.
5366
```bash
5467
npx wrangler login
5568
```
5669

57-
## 2. Create an R2 bucket
70+
## 2. Create an R2 bucket and enable R2 Data Catalog
71+
72+
<Tabs syncKey='CLIvDash'>
73+
<TabItem label='Wrangler CLI'>
74+
75+
Create an R2 bucket:
76+
77+
```bash
78+
npx wrangler r2 bucket create r2-sql-demo
79+
```
80+
81+
</TabItem>
82+
<TabItem label='Dashboard'>
83+
84+
<Steps>
85+
1. In the Cloudflare dashboard, go to the **R2 object storage** page.
86+
<DashButton url="/?to=/:account/r2/overview" />
5887

59-
Create a new R2 bucket:
88+
2. Select **Create bucket**.
89+
90+
3. Enter the bucket name: r2-sql-demo
91+
92+
4. Select **Create bucket**.
93+
</Steps>
94+
</TabItem>
95+
</Tabs>
96+
97+
## 2. Enable R2 Data Catalog
98+
99+
<Tabs syncKey='CLIvDash'>
100+
<TabItem label='Wrangler CLI'>
101+
102+
Enable the catalog on your R2 bucket:
60103

61104
```bash
62-
npx wrangler r2 bucket create r2-sql-demo
105+
npx wrangler r2 bucket catalog enable r2-sql-demo
63106
```
64107

65-
## 3. Enable R2 Data Catalog
108+
When you run this command, take note of the "Warehouse". You will need these later.
109+
110+
</TabItem>
111+
<TabItem label='Dashboard'>
112+
113+
<Steps>
114+
1. In the Cloudflare dashboard, go to the **R2 object storage** page.
115+
<DashButton url="/?to=/:account/r2/overview" />
116+
117+
2. Select the bucket: r2-sql-demo.
118+
119+
3. Switch to the **Settings** tab, scroll down to **R2 Data Catalog**, and select **Enable**.
120+
121+
4. Once enabled, note the **Catalog URI** and **Warehouse name**.
122+
</Steps>
123+
</TabItem>
124+
</Tabs>
66125

67-
Enable [R2 Data Catalog](/r2/data-catalog/) feature on your bucket to use Apache Iceberg tables:
126+
127+
:::note
128+
Copy the warehouse (ACCOUNTID_BUCKETNAME) and paste it in the `export` below. We'll use it later in the tutorial.
129+
:::
68130

69131
```bash
70-
npx wrangler r2 bucket catalog enable r2-sql-demo
132+
export $WAREHOUSE= #Paste your warehouse here
71133
```
72-
## 4. Create the data Pipeline
73134

135+
## 3. Create the data Pipeline
136+
137+
<Tabs syncKey='CLIvDash'>
138+
<TabItem label='Wrangler CLI'>
74139
### 1. Create the Pipeline Stream
75140

76141
First, create a schema file called `demo_schema.json` with the following `json` schema:
@@ -157,13 +222,75 @@ npx wrangler pipelines create demo_pipeline \
157222
Note that there is a filter on this statement that will only send events where `numbers` is greater than 5
158223
:::
159224

225+
</TabItem>
226+
<TabItem label='Dashboard'>
227+
<Steps>
228+
1. In the Cloudflare dashboard, go to **Pipelines** > **Pipelines**.
229+
<DashButton url="/?to=/:account/pipelines" />
230+
231+
2. Select **Create Pipeline**.
232+
233+
3. **Connect to a Stream**:
234+
- Pipeline name: `demo`
235+
- Enable HTTP endpoint for sending data: Enabled
236+
- HTTP authentication: Disabled (default)
237+
- Select **Next**
238+
239+
4. **Define Input Schema**:
240+
- Select **JSON editor**
241+
- Copy in the schema:
242+
```json
243+
{
244+
"fields": [
245+
{"name": "user_id", "type": "int64", "required": true},
246+
{"name": "payload", "type": "string", "required": false},
247+
{"name": "numbers", "type": "int32", "required": false}
248+
]
249+
}
250+
```
251+
252+
- Select **Next**
253+
254+
5. **Define Sink**:
255+
- Select your R2 bucket: `r2-sql-demo`
256+
- Storage type: **R2 Data Catalog**
257+
- Namespace: `fraud_detection`
258+
- Table name: `transactions`
259+
- **Advanced Settings**: Change **Maximum Time Interval** to `30 seconds`
260+
- Select **Next**
261+
262+
6. **Credentials**:
263+
- Disable **Automatically create an Account API token for your sink**
264+
- Enter **Catalog Token** from step 1
265+
- Select **Next**
266+
267+
7. **Pipeline Definition**:
268+
- Leave the default SQL query:
269+
```sql
270+
INSERT INTO demo_sink SELECT * FROM demo_stream;
271+
```
272+
- Select **Create Pipeline**
273+
274+
8. :::note
275+
Note the **HTTP Ingest Endpoint URL** from the output. This is the endpoint you'll use to send data to your pipeline.
276+
:::
277+
278+
</Steps>
279+
280+
```bash
281+
# The http ingest endpoint
282+
export STREAM_ENDPOINT= #the http ingest endpoint from the output (see example below)
283+
```
284+
</TabItem>
285+
</Tabs>
286+
287+
160288
## 5. Send some data
161289

162290
Next, let's send some events to our stream:
163291

164292
```curl
165293
curl -X POST "$STREAM_ENDPOINT" \
166-
-H "Authorization: Bearer YOUR_API_TOKEN" \
167294
-H "Content-Type: application/json" \
168295
-d '[
169296
{
@@ -194,7 +321,7 @@ This will send 4 events in one `POST`. Since our Pipeline is filtering out recor
194321
After you've sent your events to the stream, it will take about 30 seconds for the data to show in the table since that's what we configured our `roll interval` to be in the Sink.
195322

196323
```bash
197-
npx wrangler r2 sql query "SELECT * FROM demo.first_table LIMIT 10"
324+
npx wrangler r2 sql query "$WAREHOUSE" "SELECT * FROM demo.first_table LIMIT 10"
198325
```
199326

200327
<LinkCard
@@ -205,6 +332,6 @@ npx wrangler r2 sql query "SELECT * FROM demo.first_table LIMIT 10"
205332

206333
<LinkCard
207334
title="Try another example"
208-
href="/r2/sql/tutorials/end-to-end-pipeline"
335+
href="/r2-sql/tutorials/end-to-end-pipeline"
209336
description="Detailed tutorial for setting up a simple fruad detection data pipeline and generate events for it in Python."
210337
/>

src/content/docs/r2-sql/index.mdx

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
pcx_content_type: navigation
3+
title: R2 SQL
4+
sidebar:
5+
order: 7
6+
head:
7+
- tag: title
8+
content: R2 SQL
9+
description: A distributed SQL engine for R2 Data Catalog
10+
---
11+
12+
## Query Apache Iceberg tables in R2 Data Catalog Using R2 SQL
13+
14+
15+
:::note
16+
R2 SQL is in public beta, and any developer with an R2 subscription can start using it. Currently, outside of standard R2 storage and operations, you will not be billed for your use of R2 SQL. We will update [the pricing page](/r2-sql/reference/pricing) and provide at least 30 days notice before enabling billing.
17+
:::
18+
19+
R2 SQL is Cloudflare's serverless, distributed, analytics query engine for querying [Apache Iceberg](https://iceberg.apache.org/) tables stored in [R2 data catalog](https://developers.cloudflare.com/r2/data-catalog/). R2 SQL is designed to efficiently query large amounts of data by automatically utilizing file pruning, Cloudflare's distributed compute, and R2 object storage.
20+
21+
```sh
22+
❯ npx wrangler r2 sql query "3373912de3f5202317188ae01300bd6_data-catalog" \
23+
"SELECT * FROM default.transactions LIMIT 10"
24+
25+
⛅️ wrangler 4.38.0
26+
────────────────────────────────────────────────────────────────────────────
27+
▲ [WARNING] 🚧 `wrangler r2 sql query` is an open-beta command. Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose
28+
29+
30+
┌─────────────────────────────┬──────────────────────────────────────┬─────────┬──────────┬──────────────────────────────────┬───────────────┬───────────────────┬──────────┐
31+
│ __ingest_ts │ transaction_id │ user_id │ amount │ transaction_timestamp │ location │ merchant_category │ is_fraud │
32+
├─────────────────────────────┼──────────────────────────────────────┼─────────┼──────────┼──────────────────────────────────┼───────────────┼───────────────────┼──────────┤
33+
│ 2025-09-20T22:30:11.872554Z │ fdc1beed-157c-4d2d-90cf-630fdea58051 │ 1679 │ 13241.59 │ 2025-09-20T02:23:04.269988+00:00 │ NEW_YORK │ RESTAURANT │ false
34+
├─────────────────────────────┼──────────────────────────────────────┼─────────┼──────────┼──────────────────────────────────┼───────────────┼───────────────────┼──────────┤
35+
│ 2025-09-20T22:30:11.724378Z │ ea7ef106-8284-4d08-9348-ad33989b6381 │ 1279 │ 17615.79 │ 2025-09-20T02:23:04.271090+00:00 │ MIAMI │ GAS_STATION │ true
36+
├─────────────────────────────┼──────────────────────────────────────┼─────────┼──────────┼──────────────────────────────────┼───────────────┼───────────────────┼──────────┤
37+
│ 2025-09-20T22:30:11.724330Z │ afcdee4d-5c71-42be-97ec-e282b6937a8c │ 1843 │ 7311.65 │ 2025-09-20T06:23:04.267890+00:00 │ SEATTLE │ GROCERY │ true
38+
├─────────────────────────────┼──────────────────────────────────────┼─────────┼──────────┼──────────────────────────────────┼───────────────┼───────────────────┼──────────┤
39+
│ 2025-09-20T22:30:11.657007Z │ b99d14e0-dbe0-49bc-a417-0ee57f8bed99 │ 1976 │ 15228.21 │ 2025-09-16T23:23:04.269426+00:00 │ NEW_YORK │ RETAIL │ false
40+
├─────────────────────────────┼──────────────────────────────────────┼─────────┼──────────┼──────────────────────────────────┼───────────────┼───────────────────┼──────────┤
41+
│ 2025-09-20T22:30:11.656992Z │ 712cd094-ad4c-4d24-819a-0d3daaaceea1 │ 1184 │ 7570.89 │ 2025-09-20T00:23:04.269163+00:00 │ LOS_ANGELES │ RESTAURANT │ true
42+
├─────────────────────────────┼──────────────────────────────────────┼─────────┼──────────┼──────────────────────────────────┼───────────────┼───────────────────┼──────────┤
43+
│ 2025-09-20T22:30:11.656912Z │ b5a1aab3-676d-4492-92b8-aabcde6db261 │ 1196 │ 46611.25 │ 2025-09-20T16:23:04.268693+00:00 │ NEW_YORK │ RETAIL │ true
44+
├─────────────────────────────┼──────────────────────────────────────┼─────────┼──────────┼──────────────────────────────────┼───────────────┼───────────────────┼──────────┤
45+
│ 2025-09-20T22:30:11.613740Z │ 432d3976-8d89-4813-9099-ea2afa2c0e70 │ 1720 │ 21547.9 │ 2025-09-20T05:23:04.273681+00:00 │ SAN FRANCISCO │ GROCERY │ true
46+
├─────────────────────────────┼──────────────────────────────────────┼─────────┼──────────┼──────────────────────────────────┼───────────────┼───────────────────┼──────────┤
47+
│ 2025-09-20T22:30:11.532068Z │ 25e0b851-3092-4ade-842f-e3189e07d4ee │ 1562 │ 29311.54 │ 2025-09-20T05:23:04.277405+00:00 │ NEW_YORK │ RETAIL │ false
48+
├─────────────────────────────┼──────────────────────────────────────┼─────────┼──────────┼──────────────────────────────────┼───────────────┼───────────────────┼──────────┤
49+
│ 2025-09-20T22:30:11.526037Z │ 8001746d-05fe-42fe-a189-40caf81d7aa2 │ 1817 │ 15976.5 │ 2025-09-15T16:23:04.266632+00:00 │ SEATTLE │ RESTAURANT │ true
50+
└─────────────────────────────┴──────────────────────────────────────┴─────────┴──────────┴──────────────────────────────────┴───────────────┴───────────────────┴──────────┘
51+
Read 11.3 kB across 4 files from R2
52+
On average, 3.36 kB / s
53+
```
54+
55+
Create an end to end data pipeline and query your first table in R2 SQL by following [this step by step guide](/r2-sql/tutorials/end-to-end-pipeline/), learn how to create a data pipeline that takes a stream of events and automatically creates an Apache Iceberg table, making them accessible with R2 SQL.

0 commit comments

Comments
 (0)