You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/content/docs/pipelines/index.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,7 +21,7 @@ Ingest real time data streams and load into R2, using Cloudflare Pipelines.
21
21
22
22
<Plantype="paid" />
23
23
24
-
Cloudflare Pipelines lets you ingest high volumes of real time data, without managing any infrastructure. A single pipeline can ingest up to 100 MB of data per second. Ingested data is automatically batched, written to output files, and delivered to an [R2 bucket](/r2/) in your account. You can use Pipelines to build a data lake of clickstream data, or to store events from a Worker.
24
+
Cloudflare Pipelines lets you ingest high volumes of real time data, without managing any infrastructure. Ingested data is automatically batched, written to output files, and delivered to an [R2 bucket](/r2/) in your account. You can use Pipelines to build a data lake of clickstream data, or to store events from a Worker.
25
25
26
26
## Create your first pipeline
27
27
You can setup a pipeline to ingest data via HTTP, and deliver output to R2, with a single command:
Copy file name to clipboardExpand all lines: src/content/docs/workers/platform/storage-options.mdx
+43-72Lines changed: 43 additions & 72 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,79 +11,50 @@ description: Storage and database options available on Cloudflare's developer pl
11
11
12
12
import { Render, Details } from"~/components";
13
13
14
-
Cloudflare Workers support a range of storage and database options for persisting different types of data across different use-cases, from key-value stores (like [Workers KV](/kv/)) through to SQL databases (such as [D1](/d1/)). This guide describes the use-cases suited to each storage option, as well as their performance and consistency properties.
15
-
16
-
:::note[Pages Functions]
17
-
18
-
Storage options can also be used by your front-end application built with Cloudflare Pages. For more information on available storage options for Pages applications, refer to the [Pages Functions bindings documentation](/pages/functions/bindings/).
19
-
20
-
:::
21
-
22
-
Available storage and persistency products include:
23
-
24
-
-[Workers KV](#workers-kv) for key-value storage.
25
-
-[R2](#r2) for object storage, including use-cases where S3 compatible storage is required.
26
-
-[Durable Objects](#durable-objects) for transactional, globally coordinated storage.
27
-
-[D1](#d1) as a relational, SQL-based database.
28
-
-[Queues](#queues) for job queueing, batching and inter-Service (Worker to Worker) communication.
29
-
-[Hyperdrive](/hyperdrive/) for connecting to and speeding up access to existing hosted and on-premises databases.
30
-
-[Analytics Engine](/analytics/analytics-engine/) for storing and querying (using SQL) time-series data and product metrics at scale.
31
-
-[Vectorize](/vectorize/) for vector search and storing embeddings from [Workers AI](/workers-ai/).
32
-
33
-
Applications built on the Workers platform may combine one or more storage components as they grow, scale or as requirements demand.
14
+
This guide describes the storage & database products available as part of Cloudflare Workers, including recommended use-cases and best practices.
The following table highlights the performance and consistency characteristics of the primary storage offerings available to Cloudflare Workers:
22
+
Applications can build on multiple storage & database products: for example, using Workers KV for session data; R2 for large file storage, media assets and user-uploaded files; and Hyperdrive to connect to a hosted Postgres or MySQL database.
| Maximum size per value | 25 MiB | 5 TiB per object | 128 KiB per value | 10 GB per database [^5]|
50
-
| Consistency model | Eventual: updates take up to 60s to be reflected | Strong (read-after-write) [^6]| Serializable (with transactions) | Serializable (no replicas) / Causal (with replicas) |
51
-
| Supported APIs | Workers, HTTP/REST API | Workers, S3 compatible | Workers | Workers, HTTP/REST API |
26
+
Storage options can also be used by your front-end application built with Cloudflare Pages. For more information on available storage options for Pages applications, refer to the [Pages Functions bindings documentation](/pages/functions/bindings/).
52
27
53
-
</table-wrap>
28
+
:::
54
29
55
-
[^1]: Free accounts are limited to 1 GiB of KV storage.
56
-
[^2]: Free accounts are limited to 10 GB of R2 storage.
57
-
[^3]: Free accounts are limited to 5 GB of storage for SQLite-backed Durable Objects. 50 GB limit applies for KV-backed Durable Objects. Refer to [Durable Objects limits](/durable-objects/platform/limits/).
58
-
[^4]: Free accounts are limited to 5 GB of database storage.
59
-
[^5]: Free accounts are limited to 500 MB per database.
60
-
[^6]: Refer to the [R2 documentation](/r2/reference/consistency/) for more details on R2's consistency model.
30
+
## SQL database options
61
31
62
-
<Detailsheader="Footnotes"open={true}>
63
-
1. Free accounts are limited to 1 GiB of KV storage.
32
+
There are three options for SQL-based databases available when building applications with Workers.
64
33
65
-
2. Free accounts are limited to 10 GB of R2 storage.
34
+
***Hyperdrive** if you have an existing Postgres or MySQL database, require large (1TB, 100TB or more) single databases, and/or want to use your existing database tools. You can also connect Hyperdrive to database platforms like [PlanetScale](https://planetscale.com/) or [Neon](https://neon.tech/).
35
+
***D1** for lightweight, serverless applications that are read-heavy, have global users that benefit from D1's [read replication](/d1/best-practices/read-replication/), and do not require you to manage and maintain a traditional RDBMS. You can connect to
36
+
***Durable Objects** for stateful serverless workloads, per-user or per-customer SQL state, and building distributed systems (D1 and Queues are built on Durable Objects) where Durable Object's [strict serializability](https://blog.cloudflare.com/durable-objects-easy-fast-correct-choose-three/) enables global ordering of requests and storage operations.
66
37
67
-
3. Free accounts are limited to 5 GB of storage for SQLite-backed Durable Objects. 50 GB limit applies for KV-backed Durable Objects. Refer to [Durable Objects limits](/durable-objects/platform/limits/).
38
+
### Session storage
68
39
69
-
4. Free accounts are limited to 5 GB of database storage.
40
+
We recommend using [Workers KV](/kv/) for storing session data, credentials (API keys), and/or configuration data. These are typically read at high rates (thousands of RPS or more), are not typically modified (within KV's 1 write RPS per unique key limit), and do not need to be immediately consistent.
70
41
71
-
5. Free accounts are limited to 500 MB per database.
42
+
Frequently read keys benefit from KV's [internal cache](/kv/concepts/how-kv-works/), and repeated reads to these "hot" keys will typically see latencies in the 500µs to 10ms range.
72
43
73
-
6. Refer to the [R2 documentation](/r2/reference/consistency/) for more details on R2's consistency model.
74
-
</Details>
44
+
Authentication frameworks like [OpenAuth](https://openauth.js.org/docs/storage/cloudflare/) use Workers KV as session storage when deployed to Cloudflare, and [Cloudflare Access](/cloudflare-one/policies/access/) uses KV to securely store and distribute user credentials so that they can be validated as close to the user as possible and reduce overall latency.
75
45
76
-
<Renderfile="limits_increase" />
46
+
## Product overviews
77
47
78
-
## Workers KV
48
+
###Workers KV
79
49
80
50
Workers KV is an eventually consistent key-value data store that caches on the Cloudflare global network.
81
51
82
52
It is ideal for projects that require:
83
53
84
54
- High volumes of reads and/or repeated reads to the same keys.
55
+
- Low-latency global reads (typically within 10ms for hot keys)
R2 is S3-compatible blob storage that allows developers to store large amounts of unstructured data without egress fees associated with typical cloud storage services.
98
69
@@ -109,7 +80,7 @@ To get started with R2:
109
80
- Learn about R2 [Limits](/r2/platform/limits/).
110
81
- Review the [R2 Workers API](/r2/api/workers/workers-api-reference/).
111
82
112
-
## Durable Objects
83
+
###Durable Objects
113
84
114
85
Durable Objects provide low-latency coordination and consistent storage for the Workers platform through global uniqueness and a transactional storage API.
115
86
@@ -130,7 +101,7 @@ To get started with Durable Objects:
130
101
- Get started with [Durable Objects](/durable-objects/get-started/).
131
102
- Learn about Durable Objects [Limits](/durable-objects/platform/limits/).
132
103
133
-
## D1
104
+
###D1
134
105
135
106
[D1](/d1/) is Cloudflare’s native serverless database. With D1, you can create a database by importing data or defining your tables and writing your queries within a Worker or through the API.
136
107
@@ -150,7 +121,7 @@ To get started with D1:
150
121
If your working data size exceeds 10 GB (the maximum size for a D1 database), consider splitting the database into multiple, smaller D1 databases.
151
122
:::
152
123
153
-
## Queues
124
+
###Queues
154
125
155
126
Cloudflare Queues allows developers to send and receive messages with guaranteed delivery. It integrates with [Cloudflare Workers](/workers) and offers at-least once delivery, message batching, and does not charge for egress bandwidth.
156
127
@@ -165,9 +136,9 @@ To get started with Queues:
165
136
-[Set up your first queue](/queues/get-started/).
166
137
- Learn more [about how Queues works](/queues/reference/how-queues-works/).
167
138
168
-
## Hyperdrive
139
+
###Hyperdrive
169
140
170
-
Hyperdrive is a service that accelerates queries you make to existing databases, making it faster to access your data from across the globe, irrespective of your users’ location.
141
+
Hyperdrive is a service that accelerates queries you make to MySQL and Postgres databases, making it faster to access your data from across the globe, irrespective of your users’ location.
171
142
172
143
Hyperdrive allows you to:
173
144
@@ -180,7 +151,22 @@ To get started with Hyperdrive:
180
151
-[Connect Hyperdrive](/hyperdrive/get-started/) to your existing database.
181
152
- Learn more [about how Hyperdrive speeds up your database queries](/hyperdrive/configuration/how-hyperdrive-works/).
182
153
183
-
## Analytics Engine
154
+
## Pipelines
155
+
156
+
Pipelines is a streaming ingestion service that allows you to ingest high volumes of real time data, without managing any infrastructure.
157
+
158
+
Pipelines allows you to:
159
+
160
+
- Ingest data at extremely high throughput (tens of thousands of records per second or more)
161
+
- Batch and write data directly to object storage, ready for querying
162
+
- (Future) Transform and aggregate data during ingestion
163
+
164
+
To get started with Pipelines:
165
+
166
+
-[Create a Pipeline](/pipelines/getting-started/) that can batch and write records to R2.
167
+
- Learn more [about how Pipelines works](/pipelines/concepts/how-pipelines-work/).
168
+
169
+
### Analytics Engine
184
170
185
171
Analytics Engine is Cloudflare's time-series and metrics database that allows you to write unlimited-cardinality analytics at scale using a built-in API to write data points from Workers and query that data using SQL directly.
186
172
@@ -199,7 +185,7 @@ To get started with Analytics Engine:
199
185
- See [an example of writing time-series data to Analytics Engine](/analytics/analytics-engine/recipes/usage-based-billing-for-your-saas-product/)
200
186
- Understand the [SQL API](/analytics/analytics-engine/sql-api/) for reading data from your Analytics Engine datasets
201
187
202
-
## Vectorize
188
+
###Vectorize
203
189
204
190
Vectorize is a globally distributed vector database that enables you to build full-stack, AI-powered applications with Cloudflare Workers and [Workers AI](/workers-ai/).
205
191
@@ -217,21 +203,6 @@ To get started with Vectorize:
D1 is a standalone, serverless database that provides a SQL API, using SQLite's SQL semantics, to store and access your relational data.
225
207
226
-
Hyperdrive is a service that lets you connect to your existing, regional PostgreSQL databases and improves database performance by optimizing them for global, scalable data access from Workers.
227
208
228
-
- If you are building a new project on Workers or are considering migrating your data, use D1.
229
-
- If you are building a Workers project with an existing PostgreSQL database, use Hyperdrive.
230
-
231
-
:::note
232
-
233
-
You cannot use D1 with Hyperdrive.
234
-
235
-
However, D1 does not need to be used with Hyperdrive because it does not have slow connection setups which would benefit from Hyperdrive's connection pooling. D1 data can also be cached within Workers using the [Cache API](/workers/runtime-apis/cache/).
Copy file name to clipboardExpand all lines: src/content/partials/kv/kv-vs-d1.mdx
+12-4Lines changed: 12 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,13 +2,21 @@
2
2
{}
3
3
---
4
4
5
-
## Workers KV vs D1
6
5
7
6
Cloudflare Workers KV provides an eventually consistent global key-value store that caches data throughout Cloudflare's network to provide
8
-
low read latency for hot reads to keys. This is ideal for storing data that is repeatedly read by your Workers, such as configuration data, user preferences, cached values, etc. Workers KV can sustain high read throughput (unlimited requests per second per key) with \<5ms latency globally for hot reads. Workers KV is eventually consistent, so writes may take up to 60 seconds to propagate through Cloudflare's network by default.
7
+
low read latency for hot reads to keys.
9
8
10
-
Cloudflare D1 provides a SQL database that supports relational data modeling and querying. D1 supports snapshot isolation consistency and is ideal for
11
-
workloads that store user data or general web application data.
9
+
* This is ideal for storing data that is repeatedly read by your Workers, such as configuration data, user preferences, cached values, etc.
10
+
* Workers KV can sustain high read throughput (unlimited requests per second per key) with \<10ms latency globally for hot reads.
11
+
* Workers KV is eventually consistent, so writes may take up to 60 seconds to propagate through Cloudflare's network by default.
12
+
13
+
Cloudflare D1 provides a SQL database that supports relational data modeling and querying.
14
+
15
+
* D1 is built on top of SQLite, and exposes a SQL interface that is supported by many ORMs.
16
+
* Built-in [read replication](/d1/best-practices/read-replication/) enables you to automatically replicate data globally whilst still maintaining strong consistency.
17
+
* D1 supports snapshot isolation consistency and is ideal for workloads that store user data or general web application data.
18
+
19
+
### Use cases
12
20
13
21
- Use Workers KV if you need to store and access configuration data that will be read by Workers frequently, is written infrequently (\<1 RPS per key) and can tolerate eventual consistency.
14
22
- Use D1 if you need to store general application data, need SQL access to your data, and require strong consistency (writes are immediately visible after being committed).
| Object storage |[R2](/r2/)| User-facing web assets, images, machine learning and training datasets, analytics datasets, log and event data. |
9
-
| SQL database |[D1](/d1/)| Relational data, including user profiles, product listings and orders, and/or customer data. |
10
-
| Time-series metrics |[Analytics Engine](/analytics/analytics-engine/)| Write and query high-cardinality time-series data, usage metrics, and service-level telemetry using Workers and/or SQL. |
11
-
| Global co-ordination |[Durable Objects](/durable-objects/)| Building collaborative applications; global co-ordination across clients; strongly consistent, transactional storage. |
12
-
| Vector search (database) |[Vectorize](/vectorize/)| Storing [embeddings](/workers-ai/models/#text-embeddings) from AI models for semantic search and classification tasks. |
| Object storage / blob storage |[R2](/r2/)| User-facing web assets, images, machine learning and training datasets, analytics datasets, log and event data. |
9
+
| Accelerate a Postgres or MySQL database |[Hyperdrive](/hyperdrive/)| Connecting to an existing database in a cloud or on-premise using your existing database drivers & ORMs. |
10
+
| Global coordination & stateful serverless |[Durable Objects](/durable-objects/)| Building collaborative applications; global coordination across clients; real-time WebSocket applications; strongly consistent, transactional storage. |
11
+
| Lightweight SQL database |[D1](/d1/)| Relational data, including user profiles, product listings and orders, and/or customer data. |
12
+
| Task processing, batching and messaging |[Queues](/queues/)| Background job processing (emails, notifications, APIs), message queuing, and deferred tasks. |
13
+
| Vector search & embeddings queries |[Vectorize](/vectorize/)| Storing [embeddings](/workers-ai/models/#text-embeddings) from AI models for semantic search and classification tasks. |
14
+
| Streaming ingestion |[Pipelines](/pipelines/)| Streaming data ingestion and processing, including clickstream analytics, telemetry/log data, and structured data for querying |
15
+
| Time-series metrics |[Analytics Engine](/analytics/analytics-engine/)| Write and query high-cardinality time-series data, usage metrics, and service-level telemetry using Workers and/or SQL. |
![hyperlint-ai[bot]](https://avatars.githubusercontent.com/in/718456?v=4&size=40){kind=avatar}
0 commit comments