diff --git a/sources/platform/storage/dataset.md b/sources/platform/storage/dataset.md index e4ce472499..01831adaeb 100644 --- a/sources/platform/storage/dataset.md +++ b/sources/platform/storage/dataset.md @@ -384,7 +384,7 @@ By default, the whole result is wrapped in an `` element, while each pag You can grant [access rights](../collaboration/index.md) to your dataset through the **Share** button under the **Actions** menu. For more details, check the [full list of permissions](../collaboration/list_of_permissions.md). -### Sharing datasets between runs {#sharing-datasets-between-runs} +### Sharing datasets between runs You can access a dataset from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run as long as you know its _name_ or _ID_. @@ -450,7 +450,7 @@ See the [Storage overview](/platform/storage/usage#sharing-storages-between-runs - The maximum length for dataset names is 63 characters. -### Rate limiting {#rate-limiting} +### Rate limiting The rate limit for pushing data to a dataset through the [API](/api/v2/dataset-items-post) is capped at _200 requests per second_ for each dataset, a measure to prevent overloading Apify servers. diff --git a/sources/platform/storage/index.md b/sources/platform/storage/index.md index 226875ec79..5682d4a8f7 100644 --- a/sources/platform/storage/index.md +++ b/sources/platform/storage/index.md @@ -8,8 +8,7 @@ slug: /storage import Card from "@site/src/components/Card"; import CardGrid from "@site/src/components/CardGrid"; - -# Storage {#storage} +import StoragePricingCalculator from "@site/src/components/StoragePricingCalculator"; **Store anything from images and key-value pairs to structured output data. Learn how to access and manage your stored data on the Apify Console or via the API.** @@ -17,6 +16,7 @@ import CardGrid from "@site/src/components/CardGrid"; The Apify platform provides three types of storage accessible both within our [Apify Console](https://console.apify.com/storage) and externally through our [REST API](/api/v2) [Apify API Clients](/api) or [SDKs](/sdk). + Never share a URL containing your authentication token, to avoid compromising your account's security.
> If the data you want to share requires a token, first download the data, then share it as a file. -### Apify API {#apify-api} +### Apify API The [Apify API](/api/v2/storage-key-value-stores) allows you to access your storages programmatically using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) and easily share your crawling results. @@ -88,7 +90,22 @@ The Apify SDKs are libraries in JavaScript or Python that provide tools for buil * JavaScript SDK requires [Node.js](https://nodejs.org/en/) 16 or later. * Python SDK requires [Python](https://www.python.org/downloads/release/python-380/) 3.8 or above. -## Rate limiting {#rate-limiting} +## Estimate your costs + +Use this tool to estimate storage costs by plan and storage type. + +
+ Estimate your storage costs + +1. Select a storage type. +1. Choose a plan. +1. Enter storage, duration, and operation counts. +1. Review the estimated total and breakdown. + + +
+ +## Rate limiting All API endpoints limit their rate of requests to protect Apify servers from overloading. The default rate limit for storage objects is _30 requests per second_. However, there are exceptions limited to _200 requests per second_ per storage object, including: @@ -112,11 +129,11 @@ If a client exceeds this limit, the API endpoints respond with the HTTP status c Go to the [API documentation](/api/v2#rate-limiting) for details and to learn what to do if you exceed the rate limit. -## Data retention {#data-retention} +## Data retention Apify securely stores your ten most recent runs indefinitely, ensuring your records are always accessible. Unnamed datasets and runs beyond the latest ten will be automatically deleted after 7 days unless otherwise specified. Named datasets are retained indefinitely. -### Preserving your storages {#preserving-storages} +### Preserving your storages To ensure indefinite retention of your storages, assign them a name. This can be done via Apify Console or through our API. First, you'll need your store's ID. You can find it in the details of the run that created it. In Apify Console, head over to your run's details and select the **Dataset**, **Key-value store**, or **Request queue** tab as appropriate. Check that store's details, and you will find its ID among them. @@ -131,7 +148,7 @@ Our SDKs and clients each have unique naming conventions for storages. For more * [SDKs](/sdk) * [API Clients](/api) -## Named and unnamed storages {#named-and-unnamed-storages} +## Named and unnamed storages The default storages for an Actor run are unnamed, identified only by an _ID_. This allows them to expire after 7 days (or longer on paid plans) conserving your storage space. If you want to preserve a storage, [assign it a name](#preserving-storages), and it will be retained indefinitely. @@ -141,11 +158,11 @@ Named and unnamed storages are identical in all aspects except for their retenti For example, storage names `janedoe~my-storage-1` and `janedoe~web-scrape-results` are easier to tell apart than the alphanumerical IDs `cAbcYOfuXemTPwnIB` and `CAbcsuZbp7JHzkw1B`. -## Sharing {#sharing} +## Sharing You can grant [access rights](../collaboration/index.md) to others Apify users to view or modify your storages. Check the [full list of permissions](../collaboration/list_of_permissions.md). -### Sharing storages between runs {#sharing-storages-between-runs} +### Sharing storages between runs Storage can be accessed from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run, provided you have its _name_ or _ID_. You can access and manage storages from other runs using the same methods or endpoints as with storages from your current run. @@ -156,7 +173,7 @@ Storage can be accessed from any [Actor](../actors/index.mdx) or [task](../actor > When multiple runs try to write data to a storage simultaneously, the order of data writing cannot be controlled. Data is written as each request is processed.
> Similar principle applies in key-value stores and request queues, when a delete request for a record precedes a read request for the same record, the read request will fail. -## Deleting storages {#deleting-storages} +## Deleting storages Named storages are only removed upon your request.
You can delete storages in the following ways: @@ -171,11 +188,11 @@ You can delete storages in the following ways: [Key-value store](/sdk/python/reference/class/KeyValueStore#drop), or [Request queue](/sdk/python/reference/class/RequestQueue#drop) class. * [JavaScript API client](/api/client/js) - using the `.delete()` method in the -[dataset](/api/client/js/reference/class/DatasetClient), -[key-value store](/api/client/js/reference/class/KeyValueStoreClient), -or [request queue](/api/client/js/reference/class/RequestQueueClient) clients. + [dataset](/api/client/js/reference/class/DatasetClient), + [key-value store](/api/client/js/reference/class/KeyValueStoreClient), + or [request queue](/api/client/js/reference/class/RequestQueueClient) clients. * [Python API client](/api/client/python) - using the `.delete()` method in the -[dataset](/api/client/python#datasetclient), -[key-value store](/api/client/python/reference/class/KeyValueStoreClient), -or [request queue](/api/client/python/reference/class/RequestQueueClient) clients. + [dataset](/api/client/python#datasetclient), + [key-value store](/api/client/python/reference/class/KeyValueStoreClient), + or [request queue](/api/client/python/reference/class/RequestQueueClient) clients. * [API](/api/v2/key-value-store-delete) using the - `Delete [store]` endpoint, where `[store]` is the type of storage you want to delete. diff --git a/src/components/StoragePricingCalculator/StoragePricingCalculator.module.css b/src/components/StoragePricingCalculator/StoragePricingCalculator.module.css new file mode 100644 index 0000000000..e8e366e485 --- /dev/null +++ b/src/components/StoragePricingCalculator/StoragePricingCalculator.module.css @@ -0,0 +1,298 @@ +.calculator { + border: 1px solid var(--color-neutral-border, #b3b8d2); + border-radius: 0.5rem; + padding: 1.5rem; + margin: 2rem 0; + background-color: var(--ifm-background-surface-color, #ffffff); + box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); +} + +.dark { + border-color: var(--color-neutral-field-border, #4a5568); + box-shadow: 0 2px 4px rgba(0, 0, 0, 0.3); +} + +.header { + text-align: center; + margin-bottom: 2rem; +} + +.header h3 { + margin: 0 0 0.5rem 0; + color: var(--ifm-heading-color); + font-size: 2.2rem; +} + +.header p { + margin: 0; + color: var(--ifm-color-content-secondary); + font-size: 1.3rem; +} + +.controls { + display: flex; + flex-direction: column; + gap: 2rem; +} + +.selectionRow { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 2rem; + align-items: stretch; +} + +.section { + display: flex; + flex-direction: column; + gap: 1rem; + height: 100%; +} + +.section h4 { + margin: 0; + color: var(--ifm-heading-color); + font-size: 1.6rem; + font-weight: 600; +} + +.radioGroup { + display: flex; + flex-direction: column; + gap: 0.75rem; + flex: 1; + justify-content: space-between; +} + +.radioOption { + display: flex; + align-items: center; + gap: 0.75rem; + padding: 1rem; + border: 1px solid var(--color-neutral-border, #e2e8f0); + border-radius: 0.375rem; + cursor: pointer; + transition: all 0.2s ease; + background-color: var(--ifm-background-surface-color, #ffffff); + min-height: 4.5rem; + flex: 1; +} + +.dark .radioOption { + border-color: var(--color-neutral-field-border, #4a5568); +} + +/* Dark mode selected state */ +.dark .radioOption:has(input[type="radio"]:checked) { + border-color: var(--ifm-color-primary); + background-color: rgba(var(--ifm-color-primary-rgb), 0.15); + box-shadow: 0 0 0 1px var(--ifm-color-primary); +} + +.radioOption:hover { + border-color: var(--ifm-color-primary); + background-color: rgba(var(--ifm-color-primary-rgb), 0.05); +} + +.radioOption input[type="radio"] { + display: none; /* Hide the radio button indicator */ +} + +/* Selected state for radio options */ +.radioOption:has(input[type="radio"]:checked) { + border-color: var(--ifm-color-primary); + background-color: rgba(var(--ifm-color-primary-rgb), 0.1); + box-shadow: 0 0 0 1px var(--ifm-color-primary); +} + +.radioContent { + display: flex; + flex-direction: column; + gap: 0.25rem; + flex: 1; +} + +.radioContent strong { + color: var(--ifm-heading-color); + font-size: 1.3rem; +} + +.radioContent span { + color: var(--ifm-color-content-secondary); + font-size: 1.2rem; + line-height: 1.4; +} + +.inputGrid { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 1rem; +} + +.inputGroup { + display: flex; + flex-direction: column; + gap: 0.5rem; +} + +.inputGroup label { + font-weight: 500; + color: var(--ifm-heading-color); + font-size: 1.3rem; +} + +.inputGroup input { + padding: 0.75rem; + border: 1px solid var(--color-neutral-border, #e2e8f0); + border-radius: 0.375rem; + font-size: 1.2rem; + background-color: var(--ifm-background-surface-color, #ffffff); + color: var(--ifm-color-content); + transition: border-color 0.2s ease; +} + +.dark .inputGroup input { + border-color: var(--color-neutral-field-border, #4a5568); +} + +.inputGroup input:focus { + outline: none; + border-color: var(--ifm-color-primary); + box-shadow: 0 0 0 3px rgba(var(--ifm-color-primary-rgb), 0.1); +} + +.inputGroupFullWidth { + display: flex; + flex-direction: column; + gap: 0.5rem; + grid-column: 1 / -1; +} + +.inputGroupFullWidth label { + font-weight: 500; + color: var(--ifm-heading-color); + font-size: 1.3rem; +} + +.inputGroupFullWidth input { + padding: 0.75rem; + border: 1px solid var(--color-neutral-border, #e2e8f0); + border-radius: 0.375rem; + font-size: 1.2rem; + background-color: var(--ifm-background-surface-color, #ffffff); + color: var(--ifm-color-content); + transition: border-color 0.2s ease; +} + +.dark .inputGroupFullWidth input { + border-color: var(--color-neutral-field-border, #4a5568); +} + +.inputGroupFullWidth input:focus { + outline: none; + border-color: var(--ifm-color-primary); + box-shadow: 0 0 0 3px rgba(var(--ifm-color-primary-rgb), 0.1); +} + +.results { + margin-top: 2rem; + padding: 1.5rem; + background-color: rgba(var(--ifm-color-primary-rgb), 0.05); + border-radius: 0.5rem; + border: 1px solid rgba(var(--ifm-color-primary-rgb), 0.2); +} + +.results h4 { + margin: 0 0 1rem 0; + color: var(--ifm-heading-color); + font-size: 1.6rem; + font-weight: 600; +} + +.costBreakdown { + display: flex; + flex-direction: column; + gap: 0.75rem; +} + +.costItem { + display: flex; + justify-content: space-between; + align-items: center; + padding: 0.5rem 0; + border-bottom: 1px solid rgba(var(--ifm-color-primary-rgb), 0.1); + font-size: 1.3rem; +} + +.costItem:last-of-type { + border-bottom: none; +} + +.costItem span:first-child { + color: var(--ifm-color-content); +} + +.costItem span:last-child { + font-weight: 500; + color: var(--ifm-color-primary); +} + +.totalCost { + display: flex; + justify-content: space-between; + align-items: center; + padding: 1rem 0; + margin-top: 0.5rem; + border-top: 2px solid var(--ifm-color-primary); + font-size: 1.6rem; +} + +.totalCost strong { + color: var(--ifm-heading-color); +} + +.totalCost strong:last-child { + color: var(--ifm-color-primary); + font-size: 1.8rem; +} + +/* Admonition spacing */ +.calculator :global(.theme-admonition) { + margin: 1rem 0; +} + +/* Responsive design */ +@media (max-width: 768px) { + .calculator { + padding: 1rem; + } + + .selectionRow { + grid-template-columns: 1fr; + gap: 1.5rem; + } + + .inputGrid { + grid-template-columns: 1fr; + } + + .radioOption { + padding: 0.75rem; + } + + .results { + padding: 1rem; + } + + .costItem { + flex-direction: column; + align-items: flex-start; + gap: 0.25rem; + } + + .totalCost { + flex-direction: column; + align-items: flex-start; + gap: 0.5rem; + } +} diff --git a/src/components/StoragePricingCalculator/index.jsx b/src/components/StoragePricingCalculator/index.jsx new file mode 100644 index 0000000000..46a71aa8dc --- /dev/null +++ b/src/components/StoragePricingCalculator/index.jsx @@ -0,0 +1,239 @@ +import { useColorMode } from '@docusaurus/theme-common'; +import Admonition from '@theme/Admonition'; +import clsx from 'clsx'; +import React, { useMemo, useState } from 'react'; + +import styles from './StoragePricingCalculator.module.css'; + +const pricingTiers = [ + { + name: 'Free/Starter', + description: '$0/month & $39/month', + storageMultiplier: 1.0, + readMultiplier: 1.0, + writeMultiplier: 1.0, + listMultiplier: 1.0, + }, + { + name: 'Scale', + description: '$199/month', + storageMultiplier: 0.9, + readMultiplier: 0.9, + writeMultiplier: 0.9, + listMultiplier: 0.9, + }, + { + name: 'Business', + description: '$999/month', + storageMultiplier: 0.8, + readMultiplier: 0.8, + writeMultiplier: 0.8, + listMultiplier: 0.8, + }, +]; + +const storageTypes = [ + { + name: 'Dataset', + description: 'Stores results from web scraping and data processing', + baseStoragePrice: 1.00, + baseReadPrice: 0.0004, + baseWritePrice: 0.005, + hasLists: false, + }, + { + name: 'Key-value Store', + description: 'Stores various data types like JSON, HTML, images, and strings', + baseStoragePrice: 1.00, + baseReadPrice: 0.005, + baseWritePrice: 0.05, + baseListPrice: 0.05, + hasLists: true, + }, + { + name: 'Request Queue', + description: 'Manages URL processing for web crawling and other tasks', + baseStoragePrice: 4.00, + baseReadPrice: 0.004, + baseWritePrice: 0.02, + hasLists: false, + }, +]; + +const StoragePricingCalculator = () => { + const { isDarkTheme } = useColorMode(); + const [selectedStorageType, setSelectedStorageType] = useState(storageTypes[0]); + const [selectedTier, setSelectedTier] = useState(pricingTiers[0]); + const [storageGB, setStorageGB] = useState(1); + const [storageHours, setStorageHours] = useState(24); + const [reads, setReads] = useState(1000); + const [writes, setWrites] = useState(1000); + const [lists, setLists] = useState(1000); + + const calculations = useMemo(() => { + const storageCost = ((storageGB * storageHours) / 1000) * selectedStorageType.baseStoragePrice * selectedTier.storageMultiplier; + const readCost = (reads / 1000) * selectedStorageType.baseReadPrice * selectedTier.readMultiplier; + const writeCost = (writes / 1000) * selectedStorageType.baseWritePrice * selectedTier.writeMultiplier; + const listCost = selectedStorageType.hasLists && selectedStorageType.baseListPrice + ? (lists / 1000) * selectedStorageType.baseListPrice * (selectedTier.listMultiplier || 1) + : 0; + + const totalCost = storageCost + readCost + writeCost + listCost; + + return { + storageCost: storageCost.toFixed(4), + readCost: readCost.toFixed(4), + writeCost: writeCost.toFixed(4), + listCost: listCost.toFixed(4), + totalCost: totalCost.toFixed(4), + }; + }, [selectedStorageType, selectedTier, storageGB, storageHours, reads, writes, lists]); + + return ( +
+
+

Storage Pricing

+

Estimate costs for your storage usage

+
+ + + This is an estimate based on current pricing. Actual costs may vary. + + +
+
+
+

Storage Type

+
+ {storageTypes.map((type) => ( + + ))} +
+
+ +
+

Plan

+
+ {pricingTiers.map((tier) => ( + + ))} +
+
+
+ +
+

Usage

+
+
+ + setStorageGB(parseFloat(e.target.value) || 0)} + /> +
+
+ + setReads(parseInt(e.target.value, 10) || 0)} + /> +
+
+ + setStorageHours(parseInt(e.target.value, 10) || 0)} + /> +
+
+ + setWrites(parseInt(e.target.value, 10) || 0)} + /> +
+ {selectedStorageType.hasLists && ( +
+ + setLists(parseInt(e.target.value, 10) || 0)} + /> +
+ )} +
+
+
+ +
+

Estimated Costs

+
+
+ Storage ({storageGB} GB × {storageHours} hours) + ${calculations.storageCost} +
+
+ Reads ({reads.toLocaleString()} operations) + ${calculations.readCost} +
+
+ Writes ({writes.toLocaleString()} operations) + ${calculations.writeCost} +
+ {selectedStorageType.hasLists && ( +
+ Lists ({lists.toLocaleString()} operations) + ${calculations.listCost} +
+ )} +
+ Total Estimated Cost + ${calculations.totalCost} +
+
+
+
+ ); +}; + +export default StoragePricingCalculator;