-
Notifications
You must be signed in to change notification settings - Fork 15
Closed
apify/crawlee-python
#1159Labels
enhancementNew feature or request.New feature or request.t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.
Description
in _request_queue_client.py (async and sync client) there is a batch_add_requests
method. This method is calling Apify API which can return some unprocessed requests in rare cases. This can happen for example due to rate limiting when there is high load. Currently it seems that those unprocessed requests are not handled in any way and will be just ignored.
Maybe the method should at least retry to add unprocessed requests in some final batch? Maybe with some backoff?
(Some context apify/crawlee-python#1155 (comment))
vdusek
Metadata
Metadata
Assignees
Labels
enhancementNew feature or request.New feature or request.t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.