Skip to content

Commit 2f5a328

Browse files
authored
docs: add storage pricing calculator component (#1784)
add new component to calculate storage pricing style component add it to Storage usage page add link to calculator on Storage index page
1 parent 6b68e37 commit 2f5a328

File tree

6 files changed

+579
-25
lines changed

6 files changed

+579
-25
lines changed

sources/platform/storage/dataset.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -384,7 +384,7 @@ By default, the whole result is wrapped in an `<items/>` element, while each pag
384384

385385
You can grant [access rights](../collaboration/index.md) to your dataset through the **Share** button under the **Actions** menu. For more details, check the [full list of permissions](../collaboration/list_of_permissions.md).
386386

387-
### Sharing datasets between runs {#sharing-datasets-between-runs}
387+
### Sharing datasets between runs
388388

389389
You can access a dataset from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run as long as you know its _name_ or _ID_.
390390

@@ -450,7 +450,7 @@ See the [Storage overview](/platform/storage/usage#sharing-storages-between-runs
450450

451451
- The maximum length for dataset names is 63 characters.
452452

453-
### Rate limiting {#rate-limiting}
453+
### Rate limiting
454454

455455
The rate limit for pushing data to a dataset through the [API](/api/v2/dataset-items-post) is capped at _200 requests per second_ for each dataset, a measure to prevent overloading Apify servers.
456456

sources/platform/storage/index.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,15 +8,15 @@ slug: /storage
88

99
import Card from "@site/src/components/Card";
1010
import CardGrid from "@site/src/components/CardGrid";
11-
12-
# Storage {#storage}
11+
import StoragePricingCalculator from "@site/src/components/StoragePricingCalculator";
1312

1413
**Store anything from images and key-value pairs to structured output data. Learn how to access and manage your stored data on the Apify Console or via the API.**
1514

1615
---
1716

1817
The Apify platform provides three types of storage accessible both within our [Apify Console](https://console.apify.com/storage) and externally through our [REST API](/api/v2) [Apify API Clients](/api) or [SDKs](/sdk).
1918

19+
2020
<CardGrid>
2121
<Card
2222
title="Dataset"

sources/platform/storage/request_queue.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -558,7 +558,7 @@ A detailed tutorial on how to process one request queue with multiple Actor runs
558558
559559
You can grant [access rights](../collaboration/index.md) to your request queue through the **Share** button under the **Actions** menu. For more details check the [full list of permissions](../collaboration/list_of_permissions.md).
560560
561-
### Sharing request queues between runs {#sharing-request-queues-between-runs}
561+
### Sharing request queues between runs
562562
563563
You can access a request queue from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run as long as you know its _name_ or _ID_.
564564
@@ -620,7 +620,7 @@ Check out the [Storage overview](/platform/storage/usage#sharing-storages-betwee
620620
621621
- The maximum length for request queue name is 63 characters.
622622
623-
### Rate limiting {#rate-limiting}
623+
### Rate limiting
624624
625625
When managing request queues via [API](/api/v2/storage-request-queues-requests),
626626
CRUD ([add](/api/v2/request-queue-requests-post),

sources/platform/storage/usage.md

Lines changed: 36 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -6,30 +6,32 @@ category: platform
66
slug: /storage/usage
77
---
88

9+
import StoragePricingCalculator from "@site/src/components/StoragePricingCalculator";
10+
911
**Learn how to effectively use Apify's storage options. Understand key aspects of data retention, rate limiting, and secure sharing.**
1012

1113
---
1214

13-
## Dataset {#dataset}
15+
## Dataset
1416

1517
[Dataset](./dataset.md) storage allows you to store a series of data objects, such as results from web scraping, crawling, or data processing jobs. You can export your datasets in JSON, CSV, XML, RSS, Excel, or HTML formats.
1618

1719
![Dataset graphic](../images/datasets-overview.png)
1820

19-
## Key-value store {#key-value-store}
21+
## Key-value store
2022

2123
The [key-value store](./key_value_store.md) is ideal for saving data records such as files, screenshots of web pages, and PDFs or for persisting your Actor's state. The records are accessible under a unique name and can be written and read quickly.
2224

2325
![Key-value store graphic](../images/key-value-overview.svg)
2426

2527

26-
## Request queue {#request-queue}
28+
## Request queue
2729

2830
[Request queues](./request_queue.md) allow you to dynamically maintain a queue of URLs of web pages. You can use this when recursively crawling websites: you start from initial URLs and add new links as they are found while skipping duplicates.
2931

3032
![Request queue graphic](../images/request-queue-overview.svg)
3133

32-
## Basic usage {#basic-usage}
34+
## Basic usage
3335

3436
You can access your storage in several ways:
3537

@@ -38,7 +40,7 @@ You can access your storage in several ways:
3840
* [API clients](/api) - to access your storages from any Node.js/Python application.
3941
* [Apify SDKs](/sdk) - when building your own JavaScript/Python Actor.
4042

41-
### Apify Console {#apify-console}
43+
### Apify Console
4244

4345
To access your storages via Apify Console, navigate to the [**Storage**](https://console.apify.com/storage) section in the left-side menu. From there, you can click through the tabs to view your key-value stores, datasets, and request queues, and you can click on the **API** button in the top right corner to view related API endpoints. To view a storage, click its **ID**.
4446

@@ -59,7 +61,7 @@ These URLs link to API _endpoints_—the places where your data is stored. Endpo
5961
> Never share a URL containing your authentication token, to avoid compromising your account's security. <br/>
6062
> If the data you want to share requires a token, first download the data, then share it as a file.
6163
62-
### Apify API {#apify-api}
64+
### Apify API
6365

6466
The [Apify API](/api/v2/storage-key-value-stores) allows you to access your storages programmatically using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) and easily share your crawling results.
6567

@@ -88,7 +90,22 @@ The Apify SDKs are libraries in JavaScript or Python that provide tools for buil
8890
* JavaScript SDK requires [Node.js](https://nodejs.org/en/) 16 or later.
8991
* Python SDK requires [Python](https://www.python.org/downloads/release/python-380/) 3.8 or above.
9092

91-
## Rate limiting {#rate-limiting}
93+
## Estimate your costs
94+
95+
Use this tool to estimate storage costs by plan and storage type.
96+
97+
<details>
98+
<summary>Estimate your storage costs</summary>
99+
100+
1. Select a storage type.
101+
1. Choose a plan.
102+
1. Enter storage, duration, and operation counts.
103+
1. Review the estimated total and breakdown.
104+
105+
<StoragePricingCalculator />
106+
</details>
107+
108+
## Rate limiting
92109

93110
All API endpoints limit their rate of requests to protect Apify servers from overloading. The default rate limit for storage objects is _30 requests per second_. However, there are exceptions limited to _200 requests per second_ per storage object, including:
94111

@@ -112,11 +129,11 @@ If a client exceeds this limit, the API endpoints respond with the HTTP status c
112129

113130
Go to the [API documentation](/api/v2#rate-limiting) for details and to learn what to do if you exceed the rate limit.
114131

115-
## Data retention {#data-retention}
132+
## Data retention
116133

117134
Apify securely stores your ten most recent runs indefinitely, ensuring your records are always accessible. Unnamed datasets and runs beyond the latest ten will be automatically deleted after 7 days unless otherwise specified. Named datasets are retained indefinitely.
118135

119-
### Preserving your storages {#preserving-storages}
136+
### Preserving your storages
120137

121138
To ensure indefinite retention of your storages, assign them a name. This can be done via Apify Console or through our API. First, you'll need your store's ID. You can find it in the details of the run that created it. In Apify Console, head over to your run's details and select the **Dataset**, **Key-value store**, or **Request queue** tab as appropriate. Check that store's details, and you will find its ID among them.
122139

@@ -131,7 +148,7 @@ Our SDKs and clients each have unique naming conventions for storages. For more
131148
* [SDKs](/sdk)
132149
* [API Clients](/api)
133150

134-
## Named and unnamed storages {#named-and-unnamed-storages}
151+
## Named and unnamed storages
135152

136153
The default storages for an Actor run are unnamed, identified only by an _ID_. This allows them to expire after 7 days (or longer on paid plans) conserving your storage space. If you want to preserve a storage, [assign it a name](#preserving-storages), and it will be retained indefinitely.
137154

@@ -141,11 +158,11 @@ Named and unnamed storages are identical in all aspects except for their retenti
141158

142159
For example, storage names `janedoe~my-storage-1` and `janedoe~web-scrape-results` are easier to tell apart than the alphanumerical IDs `cAbcYOfuXemTPwnIB` and `CAbcsuZbp7JHzkw1B`.
143160

144-
## Sharing {#sharing}
161+
## Sharing
145162

146163
You can grant [access rights](../collaboration/index.md) to others Apify users to view or modify your storages. Check the [full list of permissions](../collaboration/list_of_permissions.md).
147164

148-
### Sharing storages between runs {#sharing-storages-between-runs}
165+
### Sharing storages between runs
149166

150167
Storage can be accessed from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run, provided you have its _name_ or _ID_. You can access and manage storages from other runs using the same methods or endpoints as with storages from your current run.
151168

@@ -156,7 +173,7 @@ Storage can be accessed from any [Actor](../actors/index.mdx) or [task](../actor
156173
> When multiple runs try to write data to a storage simultaneously, the order of data writing cannot be controlled. Data is written as each request is processed. <br/>
157174
> Similar principle applies in key-value stores and request queues, when a delete request for a record precedes a read request for the same record, the read request will fail.
158175
159-
## Deleting storages {#deleting-storages}
176+
## Deleting storages
160177

161178
Named storages are only removed upon your request.<br/>
162179
You can delete storages in the following ways:
@@ -171,11 +188,11 @@ You can delete storages in the following ways:
171188
[Key-value store](/sdk/python/reference/class/KeyValueStore#drop),
172189
or [Request queue](/sdk/python/reference/class/RequestQueue#drop) class.
173190
* [JavaScript API client](/api/client/js) - using the `.delete()` method in the
174-
[dataset](/api/client/js/reference/class/DatasetClient),
175-
[key-value store](/api/client/js/reference/class/KeyValueStoreClient),
176-
or [request queue](/api/client/js/reference/class/RequestQueueClient) clients.
191+
[dataset](/api/client/js/reference/class/DatasetClient),
192+
[key-value store](/api/client/js/reference/class/KeyValueStoreClient),
193+
or [request queue](/api/client/js/reference/class/RequestQueueClient) clients.
177194
* [Python API client](/api/client/python) - using the `.delete()` method in the
178-
[dataset](/api/client/python#datasetclient),
179-
[key-value store](/api/client/python/reference/class/KeyValueStoreClient),
180-
or [request queue](/api/client/python/reference/class/RequestQueueClient) clients.
195+
[dataset](/api/client/python#datasetclient),
196+
[key-value store](/api/client/python/reference/class/KeyValueStoreClient),
197+
or [request queue](/api/client/python/reference/class/RequestQueueClient) clients.
181198
* [API](/api/v2/key-value-store-delete) using the - `Delete [store]` endpoint, where `[store]` is the type of storage you want to delete.

0 commit comments

Comments
 (0)