Webscraping workshop - finding robots.txt

[clairecahoon]

In the first activity, you're asked to find a robots.txt file assess if you could scrape the site. I'm new to this, so I don't know where I'd look for a robots.txt file. A hint, an example, or some instructions might be helpful, unless figuring that out is part of the activity.

[csbailey5t]

This is a bit of the result of my materials not being stand-alone, but meant to accompany instruction. By that point in the workshop, I'll have shown how to get to a robots.txt file. That said, maybe a hint to google it would be sufficient.