Feature Request: Create a help page that informs councils what our scrapers are #1954
Description
Background: Scrapers crawl council websites, and we can configure the User-Agent Header to make it clear that the request is coming from Planning Alerts.
Other Web Crawlers contain a URL to let server administrators know who they are and what they do. For example:
http://www.google.com/bot.html
http://www.bing.com/bingbot.htm
http://lucene.apache.org/nutch/bot.html
https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0
We should create a page similar to the above that explains to councils:
- Who we are
- What our scrapers do
- Why they shouldn't block us!