Merge remote-tracking branch 'origin/master' into enhanced-missing-local-storage-error

Author: Pijukatel

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 All notable changes to this project will be documented in this file.
 
+## [2.4.0](https://github.com/apify/apify-sdk-python/releases/tag/v2.4.0) (2025-03-07)
+
+### 🚀 Features
+
+- Update to Crawlee v0.6 ([#420](https://github.com/apify/apify-sdk-python/pull/420)) ([9be4336](https://github.com/apify/apify-sdk-python/commit/9be433667231cc5739861fa693d7a726860d6aca)) by [@vdusek](https://github.com/vdusek)
+- Add Actor `exit_process` option ([#424](https://github.com/apify/apify-sdk-python/pull/424)) ([994c832](https://github.com/apify/apify-sdk-python/commit/994c8323b994e009db0ccdcb624891a2fef97070)) by [@vdusek](https://github.com/vdusek), closes [#396](https://github.com/apify/apify-sdk-python/issues/396), [#401](https://github.com/apify/apify-sdk-python/issues/401)
+- Upgrade websockets to v14 to adapt to library API changes ([#425](https://github.com/apify/apify-sdk-python/pull/425)) ([5f49275](https://github.com/apify/apify-sdk-python/commit/5f49275ca1177e5ba56856ffe3860f6b97bee9ee)) by [@Mantisus](https://github.com/Mantisus), closes [#325](https://github.com/apify/apify-sdk-python/issues/325)
+- Add signing of public URL ([#407](https://github.com/apify/apify-sdk-python/pull/407)) ([a865461](https://github.com/apify/apify-sdk-python/commit/a865461c703aea01d91317f4fdf38c1bedd35f00)) by [@danpoletaev](https://github.com/danpoletaev)
+
+
 ## [2.3.1](https://github.com/apify/apify-sdk-python/releases/tag/v2.3.1) (2025-02-25)
 
 ### 🐛 Bug Fixes

docs/03_concepts/12_pay_per_event.mdx

@@ -0,0 +1,46 @@
+---
+id: pay-per-event
+title: Pay-per-event monetization
+description: Monetize your Actors using the pay-per-event pricing model
+---
+
+import ActorChargeSource from '!!raw-loader!./code/actor_charge.py';
+import ConditionalActorChargeSource from '!!raw-loader!./code/conditional_actor_charge.py';
+import ApiLink from '@site/src/components/ApiLink';
+import CodeBlock from '@theme/CodeBlock';
+
+Apify provides several [pricing models](https://docs.apify.com/platform/actors/publishing/monetize) for monetizing your Actors. The most recent and most flexible one is [pay-per-event](https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event), which lets you charge your users programmatically directly from your Actor. As the name suggests, you may charge the users each time a specific event occurs, for example a call to an external API or when you return a result.
+
+To use the pay-per-event pricing model, you first need to [set it up](https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event) for your Actor in the Apify console. After that, you're free to start charging for events.
+
+## Charging for events
+
+After monetization is set in the Apify console, you can add <ApiLink to="class/Actor#charge">`Actor.charge`</ApiLink> calls to your code and start monetizing!
+
+<CodeBlock language="python">
+{ActorChargeSource}
+</CodeBlock>
+
+Then you just push your code to Apify and that's it! The SDK will even keep track of the max total charge setting for you, so you will not provide more value than what the user chose to pay for.
+
+If you need finer control over charging, you can access call <ApiLink to="class/Actor#get_charging_manager">`Actor.get_charging_manager()`</ApiLink> to access the <ApiLink to="class/ChargingManager">`ChargingManager`</ApiLink>, which can provide more detailed information - for example how many events of each type can be charged before reaching the configured limit.
+
+## Transitioning from a different pricing model
+
+When you plan to start using the pay-per-event pricing model for an Actor that is already monetized with a different pricing model, your source code will need support both pricing models during the transition period enforced by the Apify platform. Arguably the most frequent case is the transition from the pay-per-result model which utilizes the `ACTOR_MAX_PAID_DATASET_ITEMS` environment variable to prevent returning unpaid dataset items. The following is an example how to handle such scenarios. The key part is the <ApiLink to="class/ChargingManager#get_pricing_info">`ChargingManager.get_pricing_info()`</ApiLink> method which returns information about the current pricing model.
+
+<CodeBlock language="python">
+{ConditionalActorChargeSource}
+</CodeBlock>
+
+## Local development
+
+It is encouraged to test your monetization code on your machine before releasing it to the public. To tell your Actor that it should work in pay-per-event mode, pass it the `ACTOR_TEST_PAY_PER_EVENT` environment variable:
+
+```shell
+ACTOR_TEST_PAY_PER_EVENT=true python -m youractor
+```
+
+If you also wish to see a log of all the events charged throughout the run, the Apify SDK keeps a log of charged events in a so called charging dataset. Your charging dataset can be found under the `charging_log` name (unless you change your storage settings, this dataset is stored in `storage/datasets/charging_log/`). Please note that this log is not available when running the Actor in production on the Apify platform.
+
+Because pricing configuration is stored by the Apify platform, all events will have a default price of $1.

docs/03_concepts/code/actor_charge.py

@@ -0,0 +1,30 @@
+from apify import Actor
+
+
+async def main() -> None:
+    async with Actor:
+        # highlight-start
+        # Charge for a single occurence of an event
+        await Actor.charge(event_name='init')
+        # highlight-end
+
+        # Prepare some mock results
+        result = [
+            {'word': 'Lorem'},
+            {'word': 'Ipsum'},
+            {'word': 'Dolor'},
+            {'word': 'Sit'},
+            {'word': 'Amet'},
+        ]
+        # highlight-start
+        # Shortcut for charging for each pushed dataset item
+        await Actor.push_data(result, 'result-item')
+        # highlight-end
+
+        # highlight-start
+        # Or you can charge for a given number of events manually
+        await Actor.charge(
+            event_name='result-item',
+            count=len(result),
+        )
+        # highlight-end

docs/03_concepts/code/conditional_actor_charge.py

@@ -0,0 +1,18 @@
+from apify import Actor
+
+
+async def main() -> None:
+    async with Actor:
+        # Check the dataset because there might already be items
+        # if the run migrated or was restarted
+        default_dataset = await Actor.open_dataset()
+        dataset_info = await default_dataset.get_info()
+        charged_items = dataset_info.item_count if dataset_info else 0
+
+        # highlight-start
+        if Actor.get_charging_manager().get_pricing_info().is_pay_per_event:
+            # highlight-end
+            await Actor.push_data({'hello': 'world'}, 'dataset-item')
+        elif charged_items < (Actor.config.max_paid_dataset_items or 0):
+            await Actor.push_data({'hello': 'world'})
+            charged_items += 1

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "apify"
-version = "2.3.1"
+version = "2.4.0"
 description = "Apify SDK for Python"
 authors = [{ name = "Apify Technologies s.r.o.", email = "[email protected]" }]
 license = { file = "LICENSE" }
@@ -35,16 +35,14 @@ keywords = [
 ]
 dependencies = [
     "apify-client>=1.9.2",
-    "apify-shared>=1.2.1",
-    "crawlee~=0.5.0",
+    "apify-shared>=1.3.0",
+    "crawlee~=0.6.0",
     "cryptography>=42.0.0",
     "httpx>=0.27.0",
     "lazy-object-proxy>=1.10.0",
     "more_itertools>=10.2.0",
     "typing-extensions>=4.1.0",
-    # TODO: Relax the upper bound once the issue is resolved:
-    # https://github.com/apify/apify-sdk-python/issues/325
-    "websockets>=10.0,<14.0.0",
+    "websockets>=14.0",
 ]
 
 [project.optional-dependencies]
@@ -62,7 +60,7 @@ scrapy = ["scrapy>=2.11.0"]
 dev = [
     "build~=1.2.0",
     "filelock~=3.17.0",
-    "griffe~=1.5.0",
+    "griffe~=1.6.0",
     "mypy~=1.15.0",
     "pre-commit~=4.1.0",
     "pydoc-markdown~=4.8.0",
@@ -74,7 +72,7 @@ dev = [
     "pytest-xdist~=3.6.0",
     "respx~=0.22.0",
     "ruff~=0.9.0",
-    "setuptools~=75.8.0",     # setuptools are used by pytest but not explicitly required
+    "setuptools~=76.0.0",     # setuptools are used by pytest but not explicitly required
 ]
 
 [tool.hatch.build.targets.wheel]

src/apify/_actor.py

@@ -4,6 +4,7 @@
 import functools
 import os
 import sys
+from contextlib import suppress
 from datetime import timedelta
 from typing import TYPE_CHECKING, Any, Callable, Literal, TypeVar, cast, overload
 
@@ -44,7 +45,7 @@
     from typing_extensions import Self
 
     from crawlee.proxy_configuration import _NewUrlFunction
-    from crawlee.storage_clients import BaseStorageClient
+    from crawlee.storage_clients import StorageClient
 
     from apify._models import Webhook
 
@@ -83,6 +84,7 @@ def __init__(
         configuration: Configuration | None = None,
         *,
         configure_logging: bool = True,
+        exit_process: bool | None = None,
     ) -> None:
         """Create an Actor instance.
 
@@ -93,7 +95,10 @@ def __init__(
             configuration: The Actor configuration to be used. If not passed, a new Configuration instance will
                 be created.
             configure_logging: Should the default logging configuration be configured?
+            exit_process: Whether the Actor should call `sys.exit` when the context manager exits. The default is
+                True except for the IPython, Pytest and Scrapy environments.
         """
+        self._exit_process = self._get_default_exit_process() if exit_process is None else exit_process
         self._is_exiting = False
 
         self._configuration = configuration or Configuration.get_global_configuration()
@@ -160,9 +165,19 @@ def __repr__(self) -> str:
 
         return super().__repr__()
 
-    def __call__(self, configuration: Configuration | None = None, *, configure_logging: bool = True) -> Self:
+    def __call__(
+        self,
+        configuration: Configuration | None = None,
+        *,
+        configure_logging: bool = True,
+        exit_process: bool | None = None,
+    ) -> Self:
         """Make a new Actor instance with a non-default configuration."""
-        return self.__class__(configuration=configuration, configure_logging=configure_logging)
+        return self.__class__(
+            configuration=configuration,
+            configure_logging=configure_logging,
+            exit_process=exit_process,
+        )
 
     @property
     def apify_client(self) -> ApifyClientAsync:
@@ -190,7 +205,7 @@ def log(self) -> logging.Logger:
         return logger
 
     @property
-    def _local_storage_client(self) -> BaseStorageClient:
+    def _local_storage_client(self) -> StorageClient:
         """The local storage client the Actor instance uses."""
         return service_locator.get_storage_client()
 
@@ -300,13 +315,7 @@ async def finalize() -> None:
         await asyncio.wait_for(finalize(), cleanup_timeout.total_seconds())
         self._is_initialized = False
 
-        if is_running_in_ipython():
-            self.log.debug(f'Not calling sys.exit({exit_code}) because Actor is running in IPython')
-        elif os.getenv('PYTEST_CURRENT_TEST', default=False):  # noqa: PLW1508
-            self.log.debug(f'Not calling sys.exit({exit_code}) because Actor is running in an unit test')
-        elif os.getenv('SCRAPY_SETTINGS_MODULE'):
-            self.log.debug(f'Not calling sys.exit({exit_code}) because Actor is running with Scrapy')
-        else:
+        if self._exit_process:
             sys.exit(exit_code)
 
     async def fail(
@@ -1150,6 +1159,26 @@ async def create_proxy_configuration(
 
         return proxy_configuration
 
+    def _get_default_exit_process(self) -> bool:
+        """Returns False for IPython, Pytest, and Scrapy environments, True otherwise."""
+        if is_running_in_ipython():
+            self.log.debug('Running in IPython, setting default `exit_process` to False.')
+            return False
+
+        # Check if running in Pytest by detecting the relevant environment variable.
+        if os.getenv('PYTEST_CURRENT_TEST'):
+            self.log.debug('Running in Pytest, setting default `exit_process` to False.')
+            return False
+
+        # Check if running in Scrapy by attempting to import it.
+        with suppress(ImportError):
+            import scrapy  # noqa: F401
+
+            self.log.debug('Running in Scrapy, setting default `exit_process` to False.')
+            return False
+
+        return True
+
 
 Actor = cast(_ActorType, Proxy(_ActorType))
 """The entry point of the SDK, through which all the Actor operations should be done."""

src/apify/_crypto.py

@@ -1,6 +1,9 @@
 from __future__ import annotations
 
 import base64
+import hashlib
+import hmac
+import string
 from typing import Any
 
 from cryptography.exceptions import InvalidTag as InvalidTagException
@@ -153,3 +156,38 @@ def decrypt_input_secrets(private_key: rsa.RSAPrivateKey, input_data: Any) -> An
                 )
 
     return input_data
+
+
+CHARSET = string.digits + string.ascii_letters
+
+
+def encode_base62(num: int) -> str:
+    """Encode the given number to base62."""
+    if num == 0:
+        return CHARSET[0]
+
+    res = ''
+    while num > 0:
+        num, remainder = divmod(num, 62)
+        res = CHARSET[remainder] + res
+    return res
+
+
+@ignore_docs
+def create_hmac_signature(secret_key: str, message: str) -> str:
+    """Generate an HMAC signature and encodes it using Base62. Base62 encoding reduces the signature length.
+
+    HMAC signature is truncated to 30 characters to make it shorter.
+
+    Args:
+        secret_key: Secret key used for signing signatures.
+        message: Message to be signed.
+
+    Returns:
+        Base62 encoded signature.
+    """
+    signature = hmac.new(secret_key.encode('utf-8'), message.encode('utf-8'), hashlib.sha256).hexdigest()[:30]
+
+    decimal_signature = int(signature, 16)
+
+    return encode_base62(decimal_signature)

src/apify/_platform_event_manager.py

@@ -4,7 +4,7 @@
 from datetime import datetime
 from typing import TYPE_CHECKING, Annotated, Any, Literal, Union
 
-import websockets.client
+import websockets.asyncio.client
 from pydantic import BaseModel, Discriminator, Field, TypeAdapter
 from typing_extensions import Self, Unpack, override
 
@@ -143,7 +143,7 @@ class PlatformEventManager(EventManager):
     but instead use it via the `Actor.on()` and `Actor.off()` methods.
     """
 
-    _platform_events_websocket: websockets.client.WebSocketClientProtocol | None = None
+    _platform_events_websocket: websockets.asyncio.client.ClientConnection | None = None
     _process_platform_messages_task: asyncio.Task | None = None
     _send_system_info_interval_task: asyncio.Task | None = None
     _connected_to_platform_websocket: asyncio.Future = asyncio.Future()
@@ -196,7 +196,7 @@ async def __aexit__(
 
     async def _process_platform_messages(self, ws_url: str) -> None:
         try:
-            async with websockets.client.connect(ws_url) as websocket:
+            async with websockets.asyncio.client.connect(ws_url) as websocket:
                 self._platform_events_websocket = websocket
                 self._connected_to_platform_websocket.set_result(True)
 

src/apify/apify_storage_client/_apify_storage_client.py

@@ -6,7 +6,7 @@
 
 from apify_client import ApifyClientAsync
 from crawlee._utils.crypto import crypto_random_object_id
-from crawlee.storage_clients import BaseStorageClient
+from crawlee.storage_clients import StorageClient
 
 from apify._utils import docs_group
 from apify.apify_storage_client._dataset_client import DatasetClient
@@ -21,7 +21,7 @@
 
 
 @docs_group('Classes')
-class ApifyStorageClient(BaseStorageClient):
+class ApifyStorageClient(StorageClient):
     """A storage client implementation based on the Apify platform storage."""
 
     def __init__(self, *, configuration: Configuration) -> None:
@@ -68,5 +68,5 @@ async def purge_on_start(self) -> None:
         pass
 
     @override
-    def get_rate_limit_errors(self) -> dict[int, int]:  # type: ignore[misc]
+    def get_rate_limit_errors(self) -> dict[int, int]:
         return self._apify_client.stats.rate_limit_errors

src/apify/apify_storage_client/_dataset_client.py

@@ -4,7 +4,7 @@
 
 from typing_extensions import override
 
-from crawlee.storage_clients._base import BaseDatasetClient
+from crawlee.storage_clients._base import DatasetClient as BaseDatasetClient
 from crawlee.storage_clients.models import DatasetItemsListPage, DatasetMetadata
 
 if TYPE_CHECKING: