Skip to content

Handle unprocessed requests in batch_add_requestsΒ #456

@Pijukatel

Description

@Pijukatel

in _request_queue_client.py (async and sync client) there is a batch_add_requests method. This method is calling Apify API which can return some unprocessed requests in rare cases. This can happen for example due to rate limiting when there is high load. Currently it seems that those unprocessed requests are not handled in any way and will be just ignored.

Maybe the method should at least retry to add unprocessed requests in some final batch? Maybe with some backoff?

(Some context apify/crawlee-python#1155 (comment))

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions