Short interval watches cause timeouts/errors in 0.40.x and later

[elixir-of-life]

Hi Guys,

I just thought I'd mention that I have a use case that involves checking around 50 watches that execute once every 1-2 minutes.

I have been using changedetection.io for about a year now in Docker running on CentOS linux.

I started having problems after upgrading to the new major release 0.40. Basically I was getting DNS resolution errors as well as timeouts. This eventually leads to every single watch that has a resolution of <10 mins failing, while the remainder of my less frequent tasks work fine - I have roughly 232 watches in all.

Forcing a recheck resolves some of the errors but not every time.

I found that once I rolled back to 0.39.22.1 branch everything is working fine again.

I added an /etc/hosts entry in the container to rule out any DNS rate limiting due to frequent requests. I could also cURL from the container and pull the pages manually just fine at all times. I also did not see any errors in the container logs while this was happened.

One thing to note as well is that when you check URLs as frequently as this, the request queue seems to block the UI which means the experience is somewhat degraded.

Anyway great project, I hope to update to 0.40 when the problems are fixed. Let me know if I can be of more assistance in debugging.

Cheers!
Ricardo