Control legitimate and protect against malicious bots

[annakrystalli]

Need a:

• robot.txt file to control legitimate crawling of site. See:
  • http://www.robotstxt.org/robotstxt.html
  • https://www.ordinarycoders.com/blog/article/robots-text-file-django
• Control annoying referrer spam (currently being pushed from binance.com)
  • Usually this would be done via .htaccess files but it seems PythonAnywhere don't expose that functionality to customers
  • Might need to add application-level blocking of certain source hosts/IPs instead.
  • Would be good to understand if pythonanywhere access logs are public or not.