Skip to content

Add support for Scrapy 2.10 and RETRY_EXCEPTIONS setting #41

@Prometheus3375

Description

@Prometheus3375

In Scrapy 2.10 RetryMiddleware was updated (PR scrapy/scrapy#5929) to deprecate EXCEPTIONS_TO_RETRY tuple. There is an error (issue scrapy/scrapy#6049) that makes RetryUserAgentMiddleware instances to throw AttributeError upon accessing EXCEPTIONS_TO_RETRY.

To resolve this problem and add support for new RETRY_EXCEPTIONS setting, __init__ must be updated in the following way:

    def __init__(self, crawler: Crawler):
        RetryMiddleware.__init__(self, crawler.settings)
        RandomUserAgentBase.__init__(self, crawler)

        if hasattr(self, 'exceptions_to_retry'):
            self.EXCEPTIONS_TO_RETRY = self.exceptions_to_retry

Such solution maintains support for previous Scrapy versions.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions