Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
a3aa0e3
WIP
Pijukatel Aug 28, 2025
be54e46
Draft for discussion.
Pijukatel Aug 29, 2025
9c6a521
Remove temp edits for e2e tests
Pijukatel Aug 29, 2025
86c4244
Simplify types
Pijukatel Aug 29, 2025
dd5f914
Update configuration according to Pydantic docs recommendation
Pijukatel Sep 1, 2025
df06b82
Properly create custom storage clients when p[assing custom configura…
Pijukatel Sep 1, 2025
430f2ad
Update the create methods
Pijukatel Sep 1, 2025
1340c5c
Add global instanc manager
Pijukatel Sep 2, 2025
a1746f3
Try to keep config in open, but rework the instance cache
Pijukatel Sep 2, 2025
e86356d
Properly set global_storage_instance_manager
Pijukatel Sep 2, 2025
07ffa19
Avoid coroutine ... was never awaited
Pijukatel Sep 2, 2025
6026b3a
Revert incorrect delete, add test that would catch it, remove useless…
Pijukatel Sep 12, 2025
330b598
Revert unintentional edit in base class
Pijukatel Sep 12, 2025
8143277
Parametrize storage_instance_manager tests by storage type
Pijukatel Sep 12, 2025
ed4bca7
Extract shared fixture and remove unused fixtures
Pijukatel Sep 12, 2025
b06054b
Update upgrading guide
Pijukatel Sep 12, 2025
fa2703b
Merge remote-tracking branch 'origin/master' into storage-clients-and…
Pijukatel Sep 15, 2025
97055ee
Apply suggestions from code review
Pijukatel Sep 16, 2025
6affd8b
Review comments
Pijukatel Sep 16, 2025
1706ed8
Renamed client_opener to client_opener_coro
Pijukatel Sep 16, 2025
6ccd4f7
Update src/crawlee/storages/_storage_instance_manager.py
Pijukatel Sep 16, 2025
043d308
Review comments
Pijukatel Sep 16, 2025
0ff47c8
Update pyproject.toml
Pijukatel Sep 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/upgrading/upgrading_to_v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,8 @@ async def main() -> None:
assert service_locator.get_storage_client() is not custom_storage_client
assert await crawler.get_dataset() is not await Dataset.open()
```
### There can be two crawlers with different services at the same time

This allows two crawlers with different services at the same time.

**Now (v1.0):**

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ dependencies = [
"protego>=0.5.0",
"psutil>=6.0.0",
"pydantic-settings>=2.2.0,!=2.7.0,!=2.7.1,!=2.8.0",
"pydantic>=2.8.0,!=2.10.0,!=2.10.1,!=2.10.2",
"pydantic>= 2.11.0",
"pyee>=9.0.0",
"tldextract>=5.1.0",
"typing-extensions>=4.1.0",
Expand Down
2 changes: 1 addition & 1 deletion src/crawlee/configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class Configuration(BaseSettings):
Settings can also be configured via environment variables, prefixed with `CRAWLEE_`.
"""

model_config = SettingsConfigDict(populate_by_name=True)
model_config = SettingsConfigDict(validate_by_name=True, validate_by_alias=True)

internal_timeout: Annotated[timedelta | None, Field(alias='crawlee_internal_timeout')] = None
"""Timeout for the internal asynchronous operations."""
Expand Down
2 changes: 1 addition & 1 deletion uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading