Skip to content

Commit 31a64c9

Browse files
danpoletaevTC-MOraethlo
authored
chore: update general resource access docs (#1988)
This PR updates general resource access documentation, more specifically: - adds more detailed information about how pre-signed URLs work - adds examples of how Actors can / should be adjusted to supported general resource access - updates [the scoped token section](https://docs.apify.com/platform/integrations/api#default-run-storages) on default storages so that it [matches the latest implementation & contents of the blog](https://blog.apify.com/p/31148a0e-c82e-4b47-85b9-64db18c8574b) Closes apify/apify-core#23249 --------- Co-authored-by: Michał Olender <[email protected]> Co-authored-by: Roman Roštár <[email protected]>
1 parent 6bbf218 commit 31a64c9

File tree

5 files changed

+186
-25
lines changed

5 files changed

+186
-25
lines changed

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,12 +104,13 @@ If the data you attempt to store in the dataset is _invalid_ (meaning any of the
104104
}
105105
```
106106

107-
The type of the AJV validation error object is [here](https://github.com/ajv-validator/ajv/blob/master/lib/types/index.ts#L86).
107+
For the complete AJV validation error object type definition, refer to the [AJV type definitions on GitHub](https://github.com/ajv-validator/ajv/blob/master/lib/types/index.ts#L86).
108108

109109
If you use the Apify JS client or Apify SDK and call `pushData` function you can access the validation errors in a `try catch` block like this:
110110

111111
<Tabs>
112112
<TabItem value="Javascript" label="Javascript" default>
113+
113114
```javascript
114115
try {
115116
const response = await Actor.pushData(items);
@@ -120,8 +121,10 @@ try {
120121
});
121122
}
122123
```
124+
123125
</TabItem>
124126
<TabItem value="Python" label="Python">
127+
125128
```python
126129
from apify import Actor
127130
from apify_client.errors import ApifyApiError
@@ -133,6 +136,7 @@ async with Actor:
133136
if 'invalidItems' in error.data:
134137
validation_errors = error.data['invalidItems']
135138
```
139+
136140
</TabItem>
137141
</Tabs>
138142

sources/platform/collaboration/general-resource-access.md

Lines changed: 172 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,9 @@ The default setting strikes a good balance for casual or internal use, but **Res
4747
You can switch to **Restricted** access at any time. If it causes issues in your workflow, you can revert to the default setting just as easily.
4848

4949
:::note Support in public Actors
50+
5051
Because this is a new setting, some existing public Actors and integrations might not support it yet. Their authors need to update them to provide a valid token on all API calls.
52+
5153
:::
5254

5355

@@ -123,27 +125,124 @@ await datasetClient.update({
123125

124126
### Sharing restricted resources with pre-signed URLs {#pre-signed-urls}
125127

126-
Even when a resource is restricted, you might still want to share it with someone outside your team — for example, to send a PDF report to a client, or include a screenshot in an automated email or Slack message. In these cases, **storage resources** (like key-value stores, datasets, and request queues) support generating **pre-signed URLs**. These are secure, time-limited links that let others access individual files without needing an Apify account or authentication.
128+
Even when a resource is restricted, you might still want to share it with someone outside your team — for example, to send a PDF report to a client, or include a screenshot in an automated email or Slack message. In these cases, _storage resources_ (like key-value stores, datasets, and request queues) support generating _pre-signed URLs_. These are secure, time-limited links that let others access individual files without needing an Apify account or authentication.
129+
130+
#### How pre-signed URLs work
131+
132+
A pre-signed URL is a regular HTTPS link that includes a cryptographic signature verifying that access has been explicitly granted by someone with valid permissions.
133+
When a pre-signed URL is used, Apify validates the signature and grants access without requiring an API token.
134+
135+
The signature can be temporary (set to expire after a specified duration) or permanent, depending on the expiration date set when it's generated.
136+
137+
#### What links can be pre-signed
138+
139+
Only selected _dataset_ and _key-value store_ endpoints support pre-signed URLs.
140+
This allows fine-grained control over what data can be shared without authentication.
141+
142+
| Resource | Link | Validity | Notes |
143+
|-----------|-----------------------|------|-------|
144+
| _Datasets_ | [Dataset items](/api/v2/dataset-items-get) (`/v2/datasets/:datasetId/items`) | Temporary or Permanent | The link provides access to all dataset items. |
145+
| _Key-value stores_ | [List of keys](/api/v2/key-value-store-keys-get) (`/v2/key-value-stores/:storeId/keys`) | Temporary or Permanent | Returns the list of keys in a store. |
146+
| _Key-value stores_ | [Single record](/api/v2/key-value-store-record-get) (`/v2/key-value-stores/:storeId/records/:recordKey`) | _Permanent only_ | The public URL for a specific record is always permanent - it stays valid as long as the record exists. |
147+
148+
:::info Automatically generated signed URLs
149+
150+
When you retrieve dataset or key-value store details using:
151+
152+
- `GET https://api.apify.com/v2/datasets/:datasetId`
153+
- `GET https://api.apify.com/v2/key-value-stores/:storeId`
154+
155+
the API response includes automatically generated fields:
156+
157+
- `itemsPublicUrl` – a pre-signed URL providing access to dataset items
158+
- `keysPublicUrl` – a pre-signed URL providing access to key-value store keys
159+
160+
These automatically generated URLs are _valid for 14 days_.
161+
162+
The response also contains:
163+
164+
- `consoleUrl` - provides a stable link to the resource's page in the Apify Console. Unlike a direct API link, Console link will prompt unauthenticated users to sign in, ensuring they have required permissions to view the resource.
165+
166+
:::
167+
168+
You can create pre-signed URLs either through the Apify Console or programmatically via the Apify API client.
169+
170+
#### How to generate pre-signed URLs in Apify Console
127171

128-
Pre-signed URLs:
172+
To generate a pre-signed link, you can use the **Export** button in Console.
129173

130-
- Work even when General resource access is restricted
131-
- Expire automatically after 14 days (by default)
132-
- Are scoped to a single resource (prevents access to other records)
133-
- Are ideal for sharing screenshots, reports, or any other one-off files
174+
:::note
134175

135-
To generate a pre-signed link, you can use the **Export** button in Console, or call the appropriate API client method.
176+
The link will include a signature _only if the general resource access is set to Restricted_. For unrestricted datasets, the link will work without a signature.
177+
178+
:::
179+
180+
##### Dataset items
181+
182+
1. Click the **Export** button.
183+
2. In the modal that appears, click **Copy shareable link**.
136184

137185
![Generating shareable link for a restricted storage resource](./images/general-resouce-access/copy-shareable-link.png)
138186

139-
:::info Console links for resources
187+
##### Key-value store records
188+
189+
1. Open a key-value store.
190+
2. Navigate to the record you want to share.
191+
3. In the **Actions** column, click the link icon to copy signed link.
140192

141-
Resource objects returned by the API and clients (like `apify-client-js`) include a `consoleUrl` property. This provides a stable link to the resource's page in the Apify Console. Unlike a direct API link, Console link will prompt unauthenticated users to sign in, ensuring they have required permissions to view the resource.
193+
![Copy pre-signed URL for KV store record](./images/general-resouce-access/copy-record-url-kv-store.png)
194+
195+
#### How to generate pre-signed URLs using Apify Client
196+
197+
You can generate pre-signed URLs programmatically for datasets and key-value stores:
198+
199+
##### Dataset items
200+
201+
```js
202+
import { ApifyClient } from "apify-client";
203+
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
204+
const datasetClient = client.dataset('my-dataset-id');
205+
206+
// Creates pre-signed URL for items (expires in 7 days)
207+
const itemsUrl = await datasetClient.createItemsPublicUrl({ expiresInSecs: 7 * 24 * 3600 });
208+
209+
// Creates permanent pre-signed URL for items
210+
const permanentItemsUrl = await datasetClient.createItemsPublicUrl();
211+
```
212+
213+
##### Key-value store list of keys
214+
215+
```js
216+
const storeClient = client.keyValueStore('my-store-id');
217+
218+
// Create pre-signed URL for list of keys (expires in 1 day)
219+
const keysPublicUrl = await storeClient.createKeysPublicUrl({ expiresInSecs: 24 * 3600 });
220+
221+
// Create permanent pre-signed URL for list of keys
222+
const permanentKeysPublicUrl = await storeClient.createKeysPublicUrl();
223+
```
142224

143-
This is ideal for use-cases like email notifications or other automated workflows.
225+
##### Key-value store record
226+
227+
```js
228+
// Get permanent URL for a single record
229+
const recordUrl = await storeClient.getRecordPublicUrl('report.pdf');
230+
```
231+
232+
:::tip Permanent signed URL
233+
234+
If the `expiresInSecs` option is not specified, the generated link will be _permanent_.
144235

145236
:::
146237

238+
#### Signing URLs manually
239+
240+
If you need finer control — for example, generating links without using Apify client — you can sign URLs manually using our reference implementation.
241+
242+
[Check the reference implementation in Apify clients](https://github.com/apify/apify-client-js/blob/5efd68a3bc78c0173a62775f79425fad78f0e6d1/src/resource_clients/dataset.ts#L179)
243+
244+
Manual signing uses standard _HMAC (SHA-256)_ with `urlSigningSecretKey` of the resource and can be easily integrated.
245+
147246
### Sharing storages by name
148247

149248
A convenient feature of storages is that you can name them. If you choose to do so there is an extra access level setting that applies to storages only, which is **Anyone with name or ID can read**. In that case anyone that knows the storage name is able to read it via API or view it using the storages Console URL.
@@ -156,19 +255,76 @@ This is very useful if you wish to expose a storage publicly with an easy to rem
156255

157256
## Implications for public Actor developers
158257

159-
If you own a public Actor in the Apify Store, you need to make sure that your Actor will work even for users who have restricted access to their resources. Over time, you might see a growing number of users with **General resource access** set to **Restricted**.
258+
If you own a public Actor in the Apify Store, you need to make sure that your Actor will work even for users who have restricted access to their resources. Over time, you might see a growing number of users with _General resource access_ set to _Restricted_.
259+
260+
In practice, this means that all API calls originating from the Actor need to have a valid API token. If you are using Apify SDK, this should be the default behavior. See the detailed guide below for more information.
261+
262+
263+
:::caution Actor runs inherit user permissions
264+
265+
Keep in mind that when users run your public Actor, the Actor makes API calls under the user account, not your developer account. This means that it follows the _General resource access_ configuration of the user account. The configuration of your developer account has no effect on the Actor users.
266+
267+
:::
268+
269+
### Migration guide to support restricted general resource access
270+
271+
This section provides a practical guide and best practices to help you update your public Actors so they fully support _Restricted general resource access_.
272+
273+
---
274+
275+
#### Always authenticate API requests
276+
277+
All API requests from your Actor should be authenticated.
278+
When using the [Apify SDK](https://docs.apify.com/sdk/js/) or [Apify Client](https://docs.apify.com/api/client/js/), this is done automatically.
279+
280+
If your Actor makes direct API calls, include the API token manually:
281+
282+
```js
283+
const response = await fetch(`https://api.apify.com/v2/key-value-stores/${storeId}`, {
284+
headers: { Authorization: `Bearer ${process.env.APIFY_TOKEN}` },
285+
});
286+
```
287+
288+
#### Generate pre-signed URLs for external sharing
289+
290+
If your Actor outputs or shares links to storages (such as datasets or key-value store records), make sure to generate pre-signed URLs instead of hardcoding API URLs.
160291

161-
:::tip Testing public access behavior
292+
For example:
293+
294+
```js
295+
import { ApifyClient } from "apify-client";
162296

163-
To test your public Actor, run it using an account with **General resource access** set to restricted. You can use your developer account, or create a temporary testing Apify account.
297+
// ❌ Avoid hardcoding raw API URLs
298+
const recordUrl = `https://api.apify.com/v2/key-value-stores/${storeId}/records/${recordKey}`;
299+
300+
// ✅ Use Apify Client methods instead
301+
const storeClient = client.keyValueStore(storeId);
302+
const recordUrl = await storeClient.getRecordPublicUrl(recordKey);
303+
304+
// Save pre-signed URL — accessible without authentication
305+
await Actor.pushData({ recordUrl });
306+
```
307+
308+
To learn more about generating pre-signed URLs, refer to the section [Sharing restricted resources with pre-signed URLs](/platform/collaboration/general-resource-access#pre-signed-urls).
309+
310+
311+
:::note Using Console URLs
312+
313+
Datasets and key-value stores also include a `consoleUrl` property.
314+
Console URLs provide stable links to the resource’s page in Apify Console.
315+
Unauthenticated users will be prompted to sign in, ensuring they have required permissions.
164316

165317
:::
166318

167-
In practice, this means that all API calls originating from the Actor need to have a valid API token. If you are using Apify SDK, this should be the default behavior.
319+
#### Test your Actor under restricted access
168320

321+
Before publishing or updating your Actor, it’s important to verify that it works correctly for users with _restricted general resource access_.
169322

170-
:::caution Actor Runs Inherit User Permissions
323+
You can easily test this by switching your own account’s setting to _Restricted_, or by creating an organization under your account and enabling restricted access there. This approach ensures your tests accurately reflect how your public Actor will behave for end users.
171324

172-
Keep in mind that when users run your public Actor, the Actor makes API calls under the user account, not your developer account. This means that it follows the **General resource access** configuration of the user account. The configuration of your developer account has no effect on the Actor users.
325+
:::tip Make sure links work as expected
326+
327+
Once you’ve enabled restricted access, run your Actor and confirm that all links generated in logs, datasets, key-value stores, and status messages remain accessible as expected. Make sure any shared URLs — especially those stored in results or notifications — work without requiring an API token.
173328

174329
:::
330+
179 KB
Loading
26.2 KB
Loading

sources/platform/integrations/programming/api.md

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -168,19 +168,20 @@ This restriction is _transitive_, which means that if the Actor runs another Act
168168

169169
When Apify [runs an Actor](/platform/actors/running/runs-and-builds#runs), it automatically creates a set of default storages (a dataset, a key-value store and request queue) that the Actor can use in runtime.
170170

171-
You can configure whether the scoped token you are going use to run the Actor should get **Write**
172-
access to these default storages.
171+
You can configure whether the scoped token you are going use to run the Actor should get access to these default storages.
173172

174173
![Configure whether the trigger token gets write access to the run default storages.](../images/api-token-scoped-default-storage-access.png)
175174

176-
:::tip
177-
Let's say your Actor produces a lot of data that you want to delete just after the Actor finishes. If you enable this toggle, your scoped token will be allowed to do that.
178-
:::
175+
If it’s **on**, the token can implicitly access the default storage of the Actor runs it triggers, or in general, of any Actor run in your account that falls within its scope. This is useful if you want to allow a third-party service to run an Actor and then read the Actor’s output (think AI agents).
179176

180-
:::caution
181-
Even if you disable this option, **the default storages can still be accessed anonymously using just their ID** (which can be obtained via the [run object](https://docs.apify.com/api/v2#tag/Actor-runsRun-object-and-its-storages)).
177+
If the toggle is **off**, the token can still trigger and inspect runs, but access to the default storages is restricted:
182178

183-
Moreover, if a scoped token can run an Actor, it can also list all its runs, including their storage IDs, ultimately exposing their content as well. If this is not desirable, change your Actor to output data into an existing named storage, or have it create a new storage.
179+
- For accounts with **Restricted general resource access**, the token cannot read or write to default storages. [Learn more about restricted general resource access](/platform/collaboration/general-resource-access).
180+
- For accounts with **Unrestricted general resource access**, the default storages can still be read anonymously using their IDs, but writing is prevented.
181+
182+
183+
:::tip
184+
Let's say your Actor produces a lot of data that you want to delete just after the Actor finishes. If you enable this toggle, your scoped token will be allowed to do that.
184185
:::
185186

186187
### Schedules

0 commit comments

Comments
 (0)