Msg From IRC:
pokechu22: https://cui.unige.ch/isi/ke/ fails the robots.txt check (
|
if 'user-agent: ia_archiver\ndisallow: /' in r.text.lower() or 'user-agent: dokuwikidumper\ndisallow: /' in r.text.lower(): |
) because it contains Disallow: /DI/cours/ImagerieNumerique/tp/ which contains Disallow: /. This seems like a bug to me