How does spider handle rate limiting and proxy rotation? #332
realonbebeto
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I have been working with the governor crate and I like how it controls requests per time interval. I am curious how spider implements that or how such configuration can be plugged in?
Similarly, in crawling how does spider handle proxy rotation and if not how can such be plugged in?
Beta Was this translation helpful? Give feedback.
All reactions