Hi,
I have found a broken robots.txt file which generates a ValueError
The line which is broken is:
Disallow: //xtbcallback.phpDisallow: //login.php
My question is should it break because the robots file is invalid or should the line be skippend and the rest of the file be read?