Skip to content

Commit 7d20865

Browse files
harshil1712harshil1712
committed
Add MotherDuck tutorial
1 parent 7158748 commit 7d20865

File tree

2 files changed

+224
-0
lines changed

2 files changed

+224
-0
lines changed

‎src/content/docs/pipelines/tutorials/index.mdx‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
type: overview
3+
pcx_content_type: navigation
4+
title: Tutorials
5+
hideChildren: true
6+
sidebar:
7+
order: 7
8+
---
9+
10+
import { GlossaryTooltip, ListTutorials } from "~/components";
11+
12+
View <GlossaryTooltip term="tutorial">tutorials</GlossaryTooltip> to help you get started with Pipelines.
13+
14+
<ListTutorials />

‎src/content/docs/pipelines/tutorials/query-data-with-motherduck/index.mdx‎

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
---
2+
updated: 2024-10-09
3+
difficulty: Intermediate
4+
content_type: 📝 Tutorial
5+
pcx_content_type: tutorial
6+
title: Query R2 data with MotherDuck
7+
products:
8+
- R2
9+
tags:
10+
- MotherDuck
11+
languages:
12+
- SQL
13+
---
14+
15+
import { Render, PackageManagers } from "~/components";
16+
17+
In this tutorial, you will learn how to ingest clickstream data to a R2 bucket using Pipelines. You will also learn how to connect the bucket to MotherDuck. You will then query the data using MotherDuck.
18+
19+
## Prerequisites
20+
21+
1. Create a [R2 bucket](/r2/buckets/create-buckets/) in your Cloudflare account.
22+
2. A [MotherDuck](https://motherduck.com/) account.
23+
24+
## 1. Create a pipeline
25+
26+
To create a new pipeline and connect it to your R2 bucket, you need the `Access Key ID` and the `Secret Access Key` of your R2 bucket. Follow the [R2 documentation](/r2/api/s3/tokens/) to get these keys. Make a note of these keys. You will need them in the next step.
27+
28+
Create a new pipeline `clickstream-pipeline` using the [wrangler CLI](/workers/wrangler/):
29+
30+
```sh
31+
npx wrangler pipelines create clickstream-pipeline --r2 <BUCKET_NAME> --access-key-id <ACCESS_KEY_ID> --secret-access-key <SECRET_ACCESS_KEY>
32+
```
33+
34+
Replace `<BUCKET_NAME>` with the name of your R2 bucket. Replace `<ACCESS_KEY_ID>` and `<SECRET_ACCESS_KEY>` with the keys you created in the previous step.
35+
36+
```output
37+
🌀 Authorizing R2 bucket <BUCKET_NAME>
38+
🌀 Creating pipeline named "clickstream-pipeline"
39+
✅ Successfully created pipeline "clickstream-pipeline" with id <PIPELINE_ID>
40+
🎉 You can now send data to your pipeline!
41+
Example: curl "https://<PIPELINE_ID>.pipelines.cloudflare.com" -d '[{"foo": "bar"}]'
42+
```
43+
44+
Make a note of the URL of your pipeline. You will need it in the next step.
45+
46+
## 2. Ingest data to R2
47+
48+
In this step, you will ingest data to your R2 bucket using `curl`. You will ingest the following JSON data to your R2 bucket:
49+
50+
<details>
51+
<summary>
52+
Click to view the JSON data
53+
</summary>
54+
```json
55+
[
56+
{
57+
"session_id": "1234567890abcdef",
58+
"user_id": "user123",
59+
"timestamp": "2024-10-08T14:30:15.123Z",
60+
"events": [
61+
{
62+
"event_id": "evt001",
63+
"event_type": "page_view",
64+
"page_url": "https://example.com/products",
65+
"timestamp": "2024-10-08T14:30:15.123Z",
66+
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
67+
"ip_address": "192.168.1.1"
68+
},
69+
{
70+
"event_id": "evt002",
71+
"event_type": "product_view",
72+
"product_id": "prod456",
73+
"page_url": "https://example.com/products/prod456",
74+
"timestamp": "2024-10-08T14:31:20.456Z"
75+
},
76+
{
77+
"event_id": "evt003",
78+
"event_type": "add_to_cart",
79+
"product_id": "prod456",
80+
"quantity": 1,
81+
"page_url": "https://example.com/products/prod456",
82+
"timestamp": "2024-10-08T14:32:05.789Z"
83+
}
84+
],
85+
"device_info": {
86+
"device_type": "desktop",
87+
"operating_system": "Windows 10",
88+
"browser": "Chrome"
89+
},
90+
"referrer": "https://google.com"
91+
},
92+
{
93+
"session_id": "abcdef1234567890",
94+
"user_id": "user456",
95+
"timestamp": "2024-10-08T15:45:30.987Z",
96+
"events": [
97+
{
98+
"event_id": "evt004",
99+
"event_type": "page_view",
100+
"page_url": "https://example.com/blog",
101+
"timestamp": "2024-10-08T15:45:30.987Z",
102+
"user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Mobile/15E148 Safari/604.1",
103+
"ip_address": "203.0.113.1"
104+
},
105+
{
106+
"event_id": "evt005",
107+
"event_type": "scroll",
108+
"scroll_depth": "75%",
109+
"page_url": "https://example.com/blog/article1",
110+
"timestamp": "2024-10-08T15:47:12.345Z"
111+
},
112+
{
113+
"event_id": "evt006",
114+
"event_type": "social_share",
115+
"platform": "twitter",
116+
"content_id": "article1",
117+
"page_url": "https://example.com/blog/article1",
118+
"timestamp": "2024-10-08T15:48:55.678Z"
119+
}
120+
],
121+
"device_info": {
122+
"device_type": "mobile",
123+
"operating_system": "iOS 14.4",
124+
"browser": "Safari"
125+
},
126+
"referrer": "https://t.co/abcd123"
127+
},
128+
{
129+
"session_id": "9876543210fedcba",
130+
"user_id": "user789",
131+
"timestamp": "2024-10-08T18:20:00.111Z",
132+
"events": [
133+
{
134+
"event_id": "evt007",
135+
"event_type": "page_view",
136+
"page_url": "https://example.com/login",
137+
"timestamp": "2024-10-08T18:20:00.111Z",
138+
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
139+
"ip_address": "198.51.100.1"
140+
},
141+
{
142+
"event_id": "evt008",
143+
"event_type": "form_submission",
144+
"form_id": "login-form",
145+
"page_url": "https://example.com/login",
146+
"timestamp": "2024-10-08T18:20:45.222Z"
147+
},
148+
{
149+
"event_id": "evt009",
150+
"event_type": "page_view",
151+
"page_url": "https://example.com/dashboard",
152+
"timestamp": "2024-10-08T18:20:50.333Z"
153+
},
154+
{
155+
"event_id": "evt010",
156+
"event_type": "feature_usage",
157+
"feature_id": "data_export",
158+
"page_url": "https://example.com/dashboard",
159+
"timestamp": "2024-10-08T18:22:30.444Z"
160+
}
161+
],
162+
"device_info": {
163+
"device_type": "desktop",
164+
"operating_system": "macOS 10.15",
165+
"browser": "Chrome"
166+
},
167+
"referrer": "https://example.com/home"
168+
}
169+
]
170+
```
171+
</details>
172+
173+
Run the following command to ingest the data to your R2 bucket using the pipeline you created in the previous step:
174+
175+
```sh
176+
curl -X POST 'https://<PIPELINE_ID>.pipelines.cloudflare.com' -d '<JSON_DATA>'
177+
```
178+
179+
Replace `<PIPELINE_ID>` with the ID of the pipeline you created in the previous step. Also, replace `<JSON_DATA>` with the JSON data provided above.
180+
181+
## 3. Connnect the R2 bucket to MotherDuck
182+
183+
In this step, you will connect the R2 bucket to MotherDuck. You can connect the bucket to MotherDuck in several ways. You can learn about these different approaches in the [MotherDuck documentation](https://motherduck.com/docs/integrations/cloud-storage/cloudflare-r2/). In this tutorial, you will connect the bucket to MotherDuck using the MotherDuck dashboard.
184+
185+
Login to the MotherDuck dashboard and click on your profile. Navigate to the **Secrets** page. Click on the **Add Secret** button and enter the following information:
186+
187+
- **Secret Name**: `Clickstream pipeline`
188+
- **Secret Type**: `Cloudflare R2`
189+
- **Access Key ID**: `ACCESS_KEY_ID` (replace with the Access Key ID you obtained in the previous step)
190+
- **Secret Access Key**: `SECRET_ACCESS_KEY` (replace with the Secret Access Key you obtained in the previous step)
191+
192+
Click on the **Add Secret** button to save the secret.
193+
194+
## 4. Query the data
195+
196+
In this step, you will query the data stored in the R2 bucket using MotherDuck. Navigate back to the MotherDuck dashboard and click on the **+** icon to add a new Notebook. Click on the **Add Cell** button to add a new cell to the notebook.
197+
198+
In the cell, enter the following query and click on the **Run** button to execute the query:
199+
200+
```sql
201+
SELECT * FROM `r2://<BUCKET_NAME>/<PATH_TO_FILE>`;
202+
```
203+
204+
Replace the `<BUCKET_NAME>` placeholder with the name of the R2 bucket you created in the previous step. Replace the `<PATH_TO_FILE>` placeholder with the path to the file you uploaded in the previous step. You can find the path to the file by navigating to the object in the Cloudflare dashboard.
205+
206+
The query will return the data stored in the R2 bucket.
207+
208+
## Conclusion
209+
210+
In this tutorial, you learned to create a pipeline and ingest data into a R2 bucket. You also learned how to connect the bucket with MotherDuck and query the data stored in the bucket. You can use this tutorial as a starting point to ingest data into an R2 bucket, and use MotherDuck to query the data stored in the bucket.

0 commit comments

Comments
 (0)