[Feature Suggestion]: per domain level waiting times

[wshyang]

Hi dgtlmoon,

I have a suggestion how you would consider it.

Oftentimes when you start your docker image after a down time, the app would immediately trigger and queue off a whole bunch of checks all at once on outstanding checks.

The result of this is that very often, if you have a whole bunch of checks from a single site, multiple requests would be generated coming from your IP towards that site at the same time.

Some sites they seem to have some sort of WAF or anti bot scrapping mechanisms in place, so if they detect behaviour that don't seem to be human they will present some sort of a CAPTCHA before displaying their content.

What I suggest is to just have an additional area to define a domain level wait time, so that when the app is processing through the queue, it simply checks and decides "When was the last time I did a check towards this domain? Can I run this check now?", and if not the check is simply skipped until the timer expires.

At the moment, my docker instance is constantly running so the situation still isn't so bad. And what I do now is when I manually added a whole bunch of pages to watch from a single site that implements such an anti scraping mechanism I'd periodically go in and just randomly do rechecks on some of them. This way, they would all have relatively random and separated out "overdue" times. It'd be nice if the app has provision for handling such cases and space out the requests accordingly too.

[dgtlmoon]

only the max number of defined workers will be checking it, not "a whole bunch"

changedetection.io/changedetectionio/__init__.py

Line 1228 in 034507f

n_workers = int(os.getenv("FETCH_WORKERS", datastore.data['settings']['requests']['workers']))

What I suggest is to just have an additional area to define a domain level wait time, so that when the app is processing through the queue, it simply checks and decides "When was the last time I did a check towards this domain? Can I run this check now?", and if not the check is simply skipped until the timer expires.

yeah maybe adds 1 minute per domain name or something

I'm open to PR's but this is very very low on the priority list atm

[wshyang]

Hi dgtlmoon,

May I know how can the number of fetch workers be configured?

[dgtlmoon]

@wshyang simply read the comment above