Commit 8721ef5
authored
feat: Add support for NDU storages (#594)
### Description
- Implement support for NDUs (non-default unnamed storages) for Apify
storage client.
### Issues
- Closes: apify/crawlee-python#1175
### Testing
- New integration tests were implemented.
### Checklist
- [x] CI passed
### Manual testing Actor
```python
import asyncio
from apify import Actor
async def main() -> None:
async with Actor:
cnt = await Actor.get_value('cnt', 0)
cnt += 1
Actor.log.info('Actor is running for the %d time', cnt)
env_dict = Actor.get_env()
env_dict = {
'id': env_dict['id'],
'build_id': env_dict['build_id'],
'default_dataset_id': env_dict['default_dataset_id'],
'default_key_value_store_id': env_dict['default_key_value_store_id'],
'default_request_queue_id': env_dict['default_request_queue_id'],
}
Actor.log.info(f'Environment variables: {env_dict}')
dataset_default = await Actor.open_dataset(force_cloud=True)
dataset_alias = await Actor.open_dataset(force_cloud=True, alias='my-alias-dataset')
dataset_alias_2 = await Actor.open_dataset(force_cloud=True, alias='my-alias-dataset-2')
dataset_named = await Actor.open_dataset(force_cloud=True, name='my-named-dataset')
Actor.log.info(f'dataset default ID: {dataset_default.id}')
Actor.log.info(f'dataset alias ID: {dataset_alias.id}')
Actor.log.info(f'dataset alias 2 ID: {dataset_alias_2.id}')
Actor.log.info(f'dataset named ID: {dataset_named.id}')
await dataset_default.push_data({'data': 'default'})
await dataset_alias.push_data({'data': 'alias'})
await dataset_alias_2.push_data({'data': 'alias 2'})
await dataset_named.push_data({'data': 'named'})
await asyncio.sleep(3)
dataset_items_default = await dataset_default.list_items()
dataset_items_alias = await dataset_alias.list_items()
dataset_items_alias_2 = await dataset_alias_2.list_items()
dataset_items_named = await dataset_named.list_items()
Actor.log.info(f'Default dataset items: {dataset_items_default}')
Actor.log.info(f'Alias dataset items: {dataset_items_alias}')
Actor.log.info(f'Alias 2 dataset items: {dataset_items_alias_2}')
Actor.log.info(f'Named dataset items: {dataset_items_named}')
if cnt < 3:
await Actor.set_value('cnt', cnt)
await Actor.reboot()
Actor.log.info('Actor is finishing...')
await asyncio.sleep(3)
env_dict = Actor.get_env()
env_dict = {
'id': env_dict['id'],
'build_id': env_dict['build_id'],
'default_dataset_id': env_dict['default_dataset_id'],
'default_key_value_store_id': env_dict['default_key_value_store_id'],
'default_request_queue_id': env_dict['default_request_queue_id'],
}
Actor.log.info(f'Environment variables: {env_dict}')
if __name__ == '__main__':
asyncio.run(main())
```
Log:
```
2025-09-16T08:15:29.454Z ACTOR: Pulling Docker image of build Cs6vcRruiN3XWMBde from registry.
2025-09-16T08:15:31.429Z ACTOR: Creating Docker container.
2025-09-16T08:15:31.614Z ACTOR: Starting Docker container.
2025-09-16T08:15:32.780Z Actor is running on the Apify platform, `disable_browser_sandbox` was changed to True.
2025-09-16T08:15:32.783Z [apify] INFO Initializing Actor...
2025-09-16T08:15:32.788Z [apify] INFO System info ({"apify_sdk_version": "2.7.1", "apify_client_version": "2.1.0", "crawlee_version": "0.6.13b37", "python_version": "3.13.7", "os": "linux"})
2025-09-16T08:15:32.919Z [apify] INFO Actor is running for the 1 time
2025-09-16T08:15:32.921Z [apify] INFO Environment variables: {'id': 'yFiEdI2cQnAwgWuWL', 'build_id': 'Cs6vcRruiN3XWMBde', 'default_dataset_id': 'dzFyI0aGwQGby34fi', 'default_key_value_store_id': '2IMIBuOc6j7OJnhf0', 'default_request_queue_id': 'e498h6IN2aTatWSoN'}
2025-09-16T08:15:33.509Z [apify] INFO dataset default ID: dzFyI0aGwQGby34fi
2025-09-16T08:15:33.511Z [apify] INFO dataset alias ID: f7fgsLCbw2wsQ46pa
2025-09-16T08:15:33.512Z [apify] INFO dataset alias 2 ID: tee4ve0yVg8VkTf5U
2025-09-16T08:15:33.514Z [apify] INFO dataset named ID: 5derRGi9fgpeknbaH
2025-09-16T08:15:37.086Z [apify] INFO Default dataset items: [{'data': 'default'}]
2025-09-16T08:15:37.087Z [apify] INFO Alias dataset items: [{'data': 'alias'}]
2025-09-16T08:15:37.089Z [apify] INFO Alias 2 dataset items: [{'data': 'alias 2'}]
2025-09-16T08:15:37.091Z [apify] INFO Named dataset items: [{'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, ... [line-too-long]
2025-09-16T08:15:37.190Z ACTOR: Actor run will reboot.
2025-09-16T08:15:37.192Z ACTOR: Sending Docker container SIGTERM signal.
2025-09-16T08:15:37.221Z ACTOR: Run was rebooted.
2025-09-16T08:15:37.222Z ACTOR: Pulling Docker image of build Cs6vcRruiN3XWMBde from registry.
2025-09-16T08:15:37.224Z ACTOR: Creating Docker container.
2025-09-16T08:15:37.368Z ACTOR: Starting Docker container.
2025-09-16T08:15:38.375Z Actor is running on the Apify platform, `disable_browser_sandbox` was changed to True.
2025-09-16T08:15:38.377Z [apify] INFO Initializing Actor...
2025-09-16T08:15:38.380Z [apify] INFO System info ({"apify_sdk_version": "2.7.1", "apify_client_version": "2.1.0", "crawlee_version": "0.6.13b37", "python_version": "3.13.7", "os": "linux"})
2025-09-16T08:15:38.504Z [apify] INFO Actor is running for the 2 time
2025-09-16T08:15:38.506Z [apify] INFO Environment variables: {'id': 'yFiEdI2cQnAwgWuWL', 'build_id': 'Cs6vcRruiN3XWMBde', 'default_dataset_id': 'dzFyI0aGwQGby34fi', 'default_key_value_store_id': '2IMIBuOc6j7OJnhf0', 'default_request_queue_id': 'e498h6IN2aTatWSoN'}
2025-09-16T08:15:39.152Z [apify] INFO dataset default ID: dzFyI0aGwQGby34fi
2025-09-16T08:15:39.154Z [apify] INFO dataset alias ID: f7fgsLCbw2wsQ46pa
2025-09-16T08:15:39.156Z [apify] INFO dataset alias 2 ID: tee4ve0yVg8VkTf5U
2025-09-16T08:15:39.158Z [apify] INFO dataset named ID: 5derRGi9fgpeknbaH
2025-09-16T08:15:42.680Z [apify] INFO Default dataset items: [{'data': 'default'}, {'data': 'default'}]
2025-09-16T08:15:42.682Z [apify] INFO Alias dataset items: [{'data': 'alias'}, {'data': 'alias'}]
2025-09-16T08:15:42.684Z [apify] INFO Alias 2 dataset items: [{'data': 'alias 2'}, {'data': 'alias 2'}]
2025-09-16T08:15:42.686Z [apify] INFO Named dataset items: [{'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, ... [line-too-long]
2025-09-16T08:15:42.788Z ACTOR: Actor run will reboot.
2025-09-16T08:15:42.790Z ACTOR: Sending Docker container SIGTERM signal.
2025-09-16T08:15:42.811Z ACTOR: Run was rebooted.
2025-09-16T08:15:42.813Z ACTOR: Pulling Docker image of build Cs6vcRruiN3XWMBde from registry.
2025-09-16T08:15:42.815Z ACTOR: Creating Docker container.
2025-09-16T08:15:42.890Z ACTOR: Starting Docker container.
2025-09-16T08:15:44.101Z Actor is running on the Apify platform, `disable_browser_sandbox` was changed to True.
2025-09-16T08:15:44.108Z [apify] INFO Initializing Actor...
2025-09-16T08:15:44.110Z [apify] INFO System info ({"apify_sdk_version": "2.7.1", "apify_client_version": "2.1.0", "crawlee_version": "0.6.13b37", "python_version": "3.13.7", "os": "linux"})
2025-09-16T08:15:44.212Z [apify] INFO Actor is running for the 3 time
2025-09-16T08:15:44.214Z [apify] INFO Environment variables: {'id': 'yFiEdI2cQnAwgWuWL', 'build_id': 'Cs6vcRruiN3XWMBde', 'default_dataset_id': 'dzFyI0aGwQGby34fi', 'default_key_value_store_id': '2IMIBuOc6j7OJnhf0', 'default_request_queue_id': 'e498h6IN2aTatWSoN'}
2025-09-16T08:15:44.535Z [apify] INFO dataset default ID: dzFyI0aGwQGby34fi
2025-09-16T08:15:44.537Z [apify] INFO dataset alias ID: f7fgsLCbw2wsQ46pa
2025-09-16T08:15:44.539Z [apify] INFO dataset alias 2 ID: tee4ve0yVg8VkTf5U
2025-09-16T08:15:44.541Z [apify] INFO dataset named ID: 5derRGi9fgpeknbaH
2025-09-16T08:15:48.067Z [apify] INFO Default dataset items: [{'data': 'default'}, {'data': 'default'}, {'data': 'default'}]
2025-09-16T08:15:48.069Z [apify] INFO Alias dataset items: [{'data': 'alias'}, {'data': 'alias'}, {'data': 'alias'}]
2025-09-16T08:15:48.071Z [apify] INFO Alias 2 dataset items: [{'data': 'alias 2'}, {'data': 'alias 2'}, {'data': 'alias 2'}]
2025-09-16T08:15:48.073Z [apify] INFO Named dataset items: [{'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, {'data': 'named'}, ... [line-too-long]
2025-09-16T08:15:48.075Z [apify] INFO Actor is finishing...
2025-09-16T08:15:51.068Z [apify] INFO Environment variables: {'id': 'yFiEdI2cQnAwgWuWL', 'build_id': 'Cs6vcRruiN3XWMBde', 'default_dataset_id': 'dzFyI0aGwQGby34fi', 'default_key_value_store_id': '2IMIBuOc6j7OJnhf0', 'default_request_queue_id': 'e498h6IN2aTatWSoN'}
2025-09-16T08:15:51.070Z [apify] INFO Exiting Actor ({"exit_code": 0})
```
Default KVS content under `__STORAGE_ALIASES_MAPPING` key:
```
{
"alias-dataset-my-alias-dataset": "f7fgsLCbw2wsQ46pa",
"alias-dataset-my-alias-dataset-2": "tee4ve0yVg8VkTf5U"
}
```1 parent 6bcbd28 commit 8721ef5
File tree
15 files changed
+577
-132
lines changed- src/apify
- events
- storage_clients
- _apify
- _file_system
- tests
- integration
- actor_source_base
- unit/storage_clients
15 files changed
+577
-132
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
401 | 401 | | |
402 | 402 | | |
403 | 403 | | |
| 404 | + | |
404 | 405 | | |
405 | 406 | | |
406 | 407 | | |
| |||
411 | 412 | | |
412 | 413 | | |
413 | 414 | | |
414 | | - | |
415 | | - | |
416 | | - | |
417 | | - | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
418 | 421 | | |
419 | 422 | | |
420 | 423 | | |
| |||
428 | 431 | | |
429 | 432 | | |
430 | 433 | | |
| 434 | + | |
431 | 435 | | |
432 | 436 | | |
433 | 437 | | |
| |||
437 | 441 | | |
438 | 442 | | |
439 | 443 | | |
| 444 | + | |
440 | 445 | | |
441 | 446 | | |
442 | 447 | | |
| |||
446 | 451 | | |
447 | 452 | | |
448 | 453 | | |
449 | | - | |
450 | | - | |
451 | | - | |
452 | | - | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
453 | 460 | | |
454 | 461 | | |
455 | 462 | | |
| |||
462 | 469 | | |
463 | 470 | | |
464 | 471 | | |
| 472 | + | |
465 | 473 | | |
466 | 474 | | |
467 | 475 | | |
| |||
471 | 479 | | |
472 | 480 | | |
473 | 481 | | |
| 482 | + | |
474 | 483 | | |
475 | 484 | | |
476 | 485 | | |
| |||
482 | 491 | | |
483 | 492 | | |
484 | 493 | | |
485 | | - | |
486 | | - | |
487 | | - | |
488 | | - | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
489 | 500 | | |
490 | 501 | | |
491 | 502 | | |
| |||
499 | 510 | | |
500 | 511 | | |
501 | 512 | | |
| 513 | + | |
502 | 514 | | |
503 | 515 | | |
504 | 516 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
15 | 17 | | |
16 | 18 | | |
17 | 19 | | |
| |||
66 | 68 | | |
67 | 69 | | |
68 | 70 | | |
| 71 | + | |
69 | 72 | | |
70 | 73 | | |
71 | 74 | | |
| |||
74 | 77 | | |
75 | 78 | | |
76 | 79 | | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
81 | 86 | | |
82 | 87 | | |
83 | | - | |
| 88 | + | |
84 | 89 | | |
85 | 90 | | |
86 | 91 | | |
87 | 92 | | |
88 | 93 | | |
89 | | - | |
90 | | - | |
91 | | - | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
92 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
93 | 101 | | |
94 | 102 | | |
95 | 103 | | |
| |||
115 | 123 | | |
116 | 124 | | |
117 | 125 | | |
118 | | - | |
119 | | - | |
120 | | - | |
| 126 | + | |
| 127 | + | |
121 | 128 | | |
122 | | - | |
123 | | - | |
124 | | - | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
125 | 142 | | |
126 | 143 | | |
127 | | - | |
| 144 | + | |
128 | 145 | | |
129 | 146 | | |
130 | 147 | | |
131 | | - | |
132 | 148 | | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
| 149 | + | |
| 150 | + | |
137 | 151 | | |
138 | | - | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
139 | 155 | | |
140 | 156 | | |
141 | 157 | | |
| |||
150 | 166 | | |
151 | 167 | | |
152 | 168 | | |
153 | | - | |
| 169 | + | |
154 | 170 | | |
155 | 171 | | |
156 | 172 | | |
| |||
Lines changed: 37 additions & 22 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| |||
58 | 59 | | |
59 | 60 | | |
60 | 61 | | |
| 62 | + | |
61 | 63 | | |
62 | 64 | | |
63 | 65 | | |
| |||
66 | 68 | | |
67 | 69 | | |
68 | 70 | | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
73 | 77 | | |
74 | 78 | | |
75 | | - | |
| 79 | + | |
76 | 80 | | |
77 | 81 | | |
78 | 82 | | |
79 | 83 | | |
80 | 84 | | |
81 | | - | |
82 | | - | |
| 85 | + | |
| 86 | + | |
83 | 87 | | |
84 | 88 | | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
85 | 92 | | |
86 | 93 | | |
87 | 94 | | |
| |||
107 | 114 | | |
108 | 115 | | |
109 | 116 | | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
117 | 133 | | |
118 | 134 | | |
119 | | - | |
| 135 | + | |
120 | 136 | | |
121 | 137 | | |
122 | 138 | | |
123 | | - | |
124 | 139 | | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
| 140 | + | |
| 141 | + | |
129 | 142 | | |
130 | | - | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
131 | 146 | | |
132 | 147 | | |
133 | 148 | | |
| |||
142 | 157 | | |
143 | 158 | | |
144 | 159 | | |
145 | | - | |
| 160 | + | |
146 | 161 | | |
147 | 162 | | |
148 | 163 | | |
| |||
0 commit comments