You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: sources/platform/storage/dataset.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -384,7 +384,7 @@ By default, the whole result is wrapped in an `<items/>` element, while each pag
384
384
385
385
You can grant [access rights](../collaboration/index.md) to your dataset through the **Share** button under the **Actions** menu. For more details, check the [full list of permissions](../collaboration/list_of_permissions.md).
386
386
387
-
### Sharing datasets between runs {#sharing-datasets-between-runs}
387
+
### Sharing datasets between runs
388
388
389
389
You can access a dataset from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run as long as you know its _name_ or _ID_.
390
390
@@ -450,7 +450,7 @@ See the [Storage overview](/platform/storage/usage#sharing-storages-between-runs
450
450
451
451
- The maximum length for dataset names is 63 characters.
452
452
453
-
### Rate limiting {#rate-limiting}
453
+
### Rate limiting
454
454
455
455
The rate limit for pushing data to a dataset through the [API](/api/v2/dataset-items-post) is capped at _200 requests per second_ for each dataset, a measure to prevent overloading Apify servers.
Copy file name to clipboardExpand all lines: sources/platform/storage/index.md
-6Lines changed: 0 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,12 +17,6 @@ import StoragePricingCalculator from "@site/src/components/StoragePricingCalcula
17
17
The Apify platform provides three types of storage accessible both within our [Apify Console](https://console.apify.com/storage) and externally through our [REST API](/api/v2)[Apify API Clients](/api) or [SDKs](/sdk).
18
18
19
19
20
-
## Pricing calculator
21
-
22
-
Use the calculator to estimate storage costs by plan and storage type. Results are estimates.
23
-
24
-
See estimates with the [pricing calculator](./usage#pricing-calculator).
Copy file name to clipboardExpand all lines: sources/platform/storage/request_queue.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -558,7 +558,7 @@ A detailed tutorial on how to process one request queue with multiple Actor runs
558
558
559
559
You can grant [access rights](../collaboration/index.md) to your request queue through the **Share** button under the **Actions**menu. For more details check the [full list of permissions](../collaboration/list_of_permissions.md).
560
560
561
-
### Sharing request queues between runs {#sharing-request-queues-between-runs}
561
+
### Sharing request queues between runs
562
562
563
563
You can access a request queue from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run as long as you know its _name_ or _ID_.
564
564
@@ -620,7 +620,7 @@ Check out the [Storage overview](/platform/storage/usage#sharing-storages-betwee
620
620
621
621
- The maximum length for request queue name is 63 characters.
622
622
623
-
### Rate limiting {#rate-limiting}
623
+
### Rate limiting
624
624
625
625
When managing request queues via [API](/api/v2/storage-request-queues-requests),
Copy file name to clipboardExpand all lines: sources/platform/storage/usage.md
+28-28Lines changed: 28 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,41 +12,26 @@ import StoragePricingCalculator from "@site/src/components/StoragePricingCalcula
12
12
13
13
---
14
14
15
-
## Pricing calculator
16
-
17
-
Use the calculator to estimate storage costs by plan and storage type. Results are estimates.
18
-
19
-
<details>
20
-
<summary>Estimate your storage costs</summary>
21
-
22
-
1. Select a storage type.
23
-
1. Choose a plan.
24
-
1. Enter storage, duration, and operation counts.
25
-
1. Review the estimated total and breakdown.
26
-
27
-
<StoragePricingCalculator />
28
-
</details>
29
-
30
-
## Dataset {#dataset}
15
+
## Dataset
31
16
32
17
[Dataset](./dataset.md) storage allows you to store a series of data objects, such as results from web scraping, crawling, or data processing jobs. You can export your datasets in JSON, CSV, XML, RSS, Excel, or HTML formats.
The [key-value store](./key_value_store.md) is ideal for saving data records such as files, screenshots of web pages, and PDFs or for persisting your Actor's state. The records are accessible under a unique name and can be written and read quickly.
39
24
40
25

41
26
42
27
43
-
## Request queue {#request-queue}
28
+
## Request queue
44
29
45
30
[Request queues](./request_queue.md) allow you to dynamically maintain a queue of URLs of web pages. You can use this when recursively crawling websites: you start from initial URLs and add new links as they are found while skipping duplicates.
@@ -55,7 +40,7 @@ You can access your storage in several ways:
55
40
*[API clients](/api) - to access your storages from any Node.js/Python application.
56
41
*[Apify SDKs](/sdk) - when building your own JavaScript/Python Actor.
57
42
58
-
### Apify Console {#apify-console}
43
+
### Apify Console
59
44
60
45
To access your storages via Apify Console, navigate to the [**Storage**](https://console.apify.com/storage) section in the left-side menu. From there, you can click through the tabs to view your key-value stores, datasets, and request queues, and you can click on the **API** button in the top right corner to view related API endpoints. To view a storage, click its **ID**.
61
46
@@ -76,7 +61,7 @@ These URLs link to API _endpoints_—the places where your data is stored. Endpo
76
61
> Never share a URL containing your authentication token, to avoid compromising your account's security. <br/>
77
62
> If the data you want to share requires a token, first download the data, then share it as a file.
78
63
79
-
### Apify API {#apify-api}
64
+
### Apify API
80
65
81
66
The [Apify API](/api/v2/storage-key-value-stores) allows you to access your storages programmatically using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) and easily share your crawling results.
82
67
@@ -105,7 +90,22 @@ The Apify SDKs are libraries in JavaScript or Python that provide tools for buil
105
90
* JavaScript SDK requires [Node.js](https://nodejs.org/en/) 16 or later.
106
91
* Python SDK requires [Python](https://www.python.org/downloads/release/python-380/) 3.8 or above.
107
92
108
-
## Rate limiting {#rate-limiting}
93
+
## Pricing calculator
94
+
95
+
Use the calculator to estimate storage costs by plan and storage type. Results are estimates.
96
+
97
+
<details>
98
+
<summary>Estimate your storage costs</summary>
99
+
100
+
1. Select a storage type.
101
+
1. Choose a plan.
102
+
1. Enter storage, duration, and operation counts.
103
+
1. Review the estimated total and breakdown.
104
+
105
+
<StoragePricingCalculator />
106
+
</details>
107
+
108
+
## Rate limiting
109
109
110
110
All API endpoints limit their rate of requests to protect Apify servers from overloading. The default rate limit for storage objects is _30 requests per second_. However, there are exceptions limited to _200 requests per second_ per storage object, including:
111
111
@@ -129,11 +129,11 @@ If a client exceeds this limit, the API endpoints respond with the HTTP status c
129
129
130
130
Go to the [API documentation](/api/v2#rate-limiting) for details and to learn what to do if you exceed the rate limit.
131
131
132
-
## Data retention {#data-retention}
132
+
## Data retention
133
133
134
134
Apify securely stores your ten most recent runs indefinitely, ensuring your records are always accessible. Unnamed datasets and runs beyond the latest ten will be automatically deleted after 7 days unless otherwise specified. Named datasets are retained indefinitely.
135
135
136
-
### Preserving your storages {#preserving-storages}
136
+
### Preserving your storages
137
137
138
138
To ensure indefinite retention of your storages, assign them a name. This can be done via Apify Console or through our API. First, you'll need your store's ID. You can find it in the details of the run that created it. In Apify Console, head over to your run's details and select the **Dataset**, **Key-value store**, or **Request queue** tab as appropriate. Check that store's details, and you will find its ID among them.
139
139
@@ -148,7 +148,7 @@ Our SDKs and clients each have unique naming conventions for storages. For more
148
148
*[SDKs](/sdk)
149
149
*[API Clients](/api)
150
150
151
-
## Named and unnamed storages {#named-and-unnamed-storages}
151
+
## Named and unnamed storages
152
152
153
153
The default storages for an Actor run are unnamed, identified only by an _ID_. This allows them to expire after 7 days (or longer on paid plans) conserving your storage space. If you want to preserve a storage, [assign it a name](#preserving-storages), and it will be retained indefinitely.
154
154
@@ -158,11 +158,11 @@ Named and unnamed storages are identical in all aspects except for their retenti
158
158
159
159
For example, storage names `janedoe~my-storage-1` and `janedoe~web-scrape-results` are easier to tell apart than the alphanumerical IDs `cAbcYOfuXemTPwnIB` and `CAbcsuZbp7JHzkw1B`.
160
160
161
-
## Sharing {#sharing}
161
+
## Sharing
162
162
163
163
You can grant [access rights](../collaboration/index.md) to others Apify users to view or modify your storages. Check the [full list of permissions](../collaboration/list_of_permissions.md).
164
164
165
-
### Sharing storages between runs {#sharing-storages-between-runs}
165
+
### Sharing storages between runs
166
166
167
167
Storage can be accessed from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run, provided you have its _name_ or _ID_. You can access and manage storages from other runs using the same methods or endpoints as with storages from your current run.
168
168
@@ -173,7 +173,7 @@ Storage can be accessed from any [Actor](../actors/index.mdx) or [task](../actor
173
173
> When multiple runs try to write data to a storage simultaneously, the order of data writing cannot be controlled. Data is written as each request is processed. <br/>
174
174
> Similar principle applies in key-value stores and request queues, when a delete request for a record precedes a read request for the same record, the read request will fail.
175
175
176
-
## Deleting storages {#deleting-storages}
176
+
## Deleting storages
177
177
178
178
Named storages are only removed upon your request.<br/>
0 commit comments