You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: sources/platform/storage/dataset.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -384,7 +384,7 @@ By default, the whole result is wrapped in an `<items/>` element, while each pag
384
384
385
385
You can grant [access rights](../collaboration/index.md) to your dataset through the **Share** button under the **Actions** menu. For more details, check the [full list of permissions](../collaboration/list_of_permissions.md).
386
386
387
-
### Sharing datasets between runs {#sharing-datasets-between-runs}
387
+
### Sharing datasets between runs
388
388
389
389
You can access a dataset from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run as long as you know its _name_ or _ID_.
390
390
@@ -450,7 +450,7 @@ See the [Storage overview](/platform/storage/usage#sharing-storages-between-runs
450
450
451
451
- The maximum length for dataset names is 63 characters.
452
452
453
-
### Rate limiting {#rate-limiting}
453
+
### Rate limiting
454
454
455
455
The rate limit for pushing data to a dataset through the [API](/api/v2/dataset-items-post) is capped at _200 requests per second_ for each dataset, a measure to prevent overloading Apify servers.
Copy file name to clipboardExpand all lines: sources/platform/storage/index.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,15 +8,15 @@ slug: /storage
8
8
9
9
import Card from "@site/src/components/Card";
10
10
import CardGrid from "@site/src/components/CardGrid";
11
-
12
-
# Storage {#storage}
11
+
import StoragePricingCalculator from "@site/src/components/StoragePricingCalculator";
13
12
14
13
**Store anything from images and key-value pairs to structured output data. Learn how to access and manage your stored data on the Apify Console or via the API.**
15
14
16
15
---
17
16
18
17
The Apify platform provides three types of storage accessible both within our [Apify Console](https://console.apify.com/storage) and externally through our [REST API](/api/v2)[Apify API Clients](/api) or [SDKs](/sdk).
Copy file name to clipboardExpand all lines: sources/platform/storage/request_queue.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -558,7 +558,7 @@ A detailed tutorial on how to process one request queue with multiple Actor runs
558
558
559
559
You can grant [access rights](../collaboration/index.md) to your request queue through the **Share** button under the **Actions**menu. For more details check the [full list of permissions](../collaboration/list_of_permissions.md).
560
560
561
-
### Sharing request queues between runs {#sharing-request-queues-between-runs}
561
+
### Sharing request queues between runs
562
562
563
563
You can access a request queue from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run as long as you know its _name_ or _ID_.
564
564
@@ -620,7 +620,7 @@ Check out the [Storage overview](/platform/storage/usage#sharing-storages-betwee
620
620
621
621
- The maximum length for request queue name is 63 characters.
622
622
623
-
### Rate limiting {#rate-limiting}
623
+
### Rate limiting
624
624
625
625
When managing request queues via [API](/api/v2/storage-request-queues-requests),
Copy file name to clipboardExpand all lines: sources/platform/storage/usage.md
+36-19Lines changed: 36 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,30 +6,32 @@ category: platform
6
6
slug: /storage/usage
7
7
---
8
8
9
+
import StoragePricingCalculator from "@site/src/components/StoragePricingCalculator";
10
+
9
11
**Learn how to effectively use Apify's storage options. Understand key aspects of data retention, rate limiting, and secure sharing.**
10
12
11
13
---
12
14
13
-
## Dataset {#dataset}
15
+
## Dataset
14
16
15
17
[Dataset](./dataset.md) storage allows you to store a series of data objects, such as results from web scraping, crawling, or data processing jobs. You can export your datasets in JSON, CSV, XML, RSS, Excel, or HTML formats.
The [key-value store](./key_value_store.md) is ideal for saving data records such as files, screenshots of web pages, and PDFs or for persisting your Actor's state. The records are accessible under a unique name and can be written and read quickly.
22
24
23
25

24
26
25
27
26
-
## Request queue {#request-queue}
28
+
## Request queue
27
29
28
30
[Request queues](./request_queue.md) allow you to dynamically maintain a queue of URLs of web pages. You can use this when recursively crawling websites: you start from initial URLs and add new links as they are found while skipping duplicates.
@@ -38,7 +40,7 @@ You can access your storage in several ways:
38
40
*[API clients](/api) - to access your storages from any Node.js/Python application.
39
41
*[Apify SDKs](/sdk) - when building your own JavaScript/Python Actor.
40
42
41
-
### Apify Console {#apify-console}
43
+
### Apify Console
42
44
43
45
To access your storages via Apify Console, navigate to the [**Storage**](https://console.apify.com/storage) section in the left-side menu. From there, you can click through the tabs to view your key-value stores, datasets, and request queues, and you can click on the **API** button in the top right corner to view related API endpoints. To view a storage, click its **ID**.
44
46
@@ -59,7 +61,7 @@ These URLs link to API _endpoints_—the places where your data is stored. Endpo
59
61
> Never share a URL containing your authentication token, to avoid compromising your account's security. <br/>
60
62
> If the data you want to share requires a token, first download the data, then share it as a file.
61
63
62
-
### Apify API {#apify-api}
64
+
### Apify API
63
65
64
66
The [Apify API](/api/v2/storage-key-value-stores) allows you to access your storages programmatically using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) and easily share your crawling results.
65
67
@@ -88,7 +90,22 @@ The Apify SDKs are libraries in JavaScript or Python that provide tools for buil
88
90
* JavaScript SDK requires [Node.js](https://nodejs.org/en/) 16 or later.
89
91
* Python SDK requires [Python](https://www.python.org/downloads/release/python-380/) 3.8 or above.
90
92
91
-
## Rate limiting {#rate-limiting}
93
+
## Estimate your costs
94
+
95
+
Use this tool to estimate storage costs by plan and storage type.
96
+
97
+
<details>
98
+
<summary>Estimate your storage costs</summary>
99
+
100
+
1. Select a storage type.
101
+
1. Choose a plan.
102
+
1. Enter storage, duration, and operation counts.
103
+
1. Review the estimated total and breakdown.
104
+
105
+
<StoragePricingCalculator />
106
+
</details>
107
+
108
+
## Rate limiting
92
109
93
110
All API endpoints limit their rate of requests to protect Apify servers from overloading. The default rate limit for storage objects is _30 requests per second_. However, there are exceptions limited to _200 requests per second_ per storage object, including:
94
111
@@ -112,11 +129,11 @@ If a client exceeds this limit, the API endpoints respond with the HTTP status c
112
129
113
130
Go to the [API documentation](/api/v2#rate-limiting) for details and to learn what to do if you exceed the rate limit.
114
131
115
-
## Data retention {#data-retention}
132
+
## Data retention
116
133
117
134
Apify securely stores your ten most recent runs indefinitely, ensuring your records are always accessible. Unnamed datasets and runs beyond the latest ten will be automatically deleted after 7 days unless otherwise specified. Named datasets are retained indefinitely.
118
135
119
-
### Preserving your storages {#preserving-storages}
136
+
### Preserving your storages
120
137
121
138
To ensure indefinite retention of your storages, assign them a name. This can be done via Apify Console or through our API. First, you'll need your store's ID. You can find it in the details of the run that created it. In Apify Console, head over to your run's details and select the **Dataset**, **Key-value store**, or **Request queue** tab as appropriate. Check that store's details, and you will find its ID among them.
122
139
@@ -131,7 +148,7 @@ Our SDKs and clients each have unique naming conventions for storages. For more
131
148
*[SDKs](/sdk)
132
149
*[API Clients](/api)
133
150
134
-
## Named and unnamed storages {#named-and-unnamed-storages}
151
+
## Named and unnamed storages
135
152
136
153
The default storages for an Actor run are unnamed, identified only by an _ID_. This allows them to expire after 7 days (or longer on paid plans) conserving your storage space. If you want to preserve a storage, [assign it a name](#preserving-storages), and it will be retained indefinitely.
137
154
@@ -141,11 +158,11 @@ Named and unnamed storages are identical in all aspects except for their retenti
141
158
142
159
For example, storage names `janedoe~my-storage-1` and `janedoe~web-scrape-results` are easier to tell apart than the alphanumerical IDs `cAbcYOfuXemTPwnIB` and `CAbcsuZbp7JHzkw1B`.
143
160
144
-
## Sharing {#sharing}
161
+
## Sharing
145
162
146
163
You can grant [access rights](../collaboration/index.md) to others Apify users to view or modify your storages. Check the [full list of permissions](../collaboration/list_of_permissions.md).
147
164
148
-
### Sharing storages between runs {#sharing-storages-between-runs}
165
+
### Sharing storages between runs
149
166
150
167
Storage can be accessed from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run, provided you have its _name_ or _ID_. You can access and manage storages from other runs using the same methods or endpoints as with storages from your current run.
151
168
@@ -156,7 +173,7 @@ Storage can be accessed from any [Actor](../actors/index.mdx) or [task](../actor
156
173
> When multiple runs try to write data to a storage simultaneously, the order of data writing cannot be controlled. Data is written as each request is processed. <br/>
157
174
> Similar principle applies in key-value stores and request queues, when a delete request for a record precedes a read request for the same record, the read request will fail.
158
175
159
-
## Deleting storages {#deleting-storages}
176
+
## Deleting storages
160
177
161
178
Named storages are only removed upon your request.<br/>
162
179
You can delete storages in the following ways:
@@ -171,11 +188,11 @@ You can delete storages in the following ways:
0 commit comments