Make `urllib.robotparser` fully support wildcard in paths

[hongyeoplee]

Some robots.txt files are not parsed at all with python internal library (those contain wildcards in paths):

• https://twitter.com/robots.txt (partially parsed)
• https://www.samsung.com/robots.txt
• https://www.koreanair.com/robots.txt

While, above files all are very well parsed with https://github.com/google/robotstxt.

Related: #114310 and #114310 (comment).

[gaogaotiantian]

Can you me more specific? What was not parsed? Could you share an example code that did not match your expectation?

[hongyeoplee]

from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url('https://www.google.com/robots.txt')
rp.read()
print(rp)
# Result #
# User-agent: AdsBot-Google
# Disallow: /maps/api/js/
# Allow: /maps/api/js
# Disallow: /maps/api/place/js/
# Disallow: /maps/api/staticmap
# ...
# Disallow: /nonprofits/account/
# Disallow: /uviewer
# Disallow: /landing/cmsnext-root/

When I run code above, there's some parsed data in rp object.
But when I set the url to Samsung (code below), nothing is printed when I print rp.

from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url('https://www.samsung.com/robots.txt')
rp.read()
print(rp)

Also, can_fetch() doesn't give me the right answer, The below shows some examples.

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://www.samsung.com/robots.txt")
rp.read()

print(rp.site_maps())                                                                    # Result : None // expected : https://www.samsung.com/sitemap.xml
print(rp.can_fetch("*", 'https://www.samsung.com/any/parking'))                          # Result : False  // expected : False
print(rp.can_fetch("*", 'https://www.samsung.com/any/parking/any'))                      # Result : False  // expected : False
print(rp.can_fetch("*", 'https://www.samsung.com/any/search/any'))                       # Result : False  // expected : False
print(rp.can_fetch("*", 'https://www.samsung.com/uk/info/contactus/email-the-ceo/'))     # Result : False  // expected : True

rp = RobotFileParser()
rp.set_url('https://www.google.com/robots.txt')
rp.read()

print(rp.can_fetch("*", 'https://www.google.com/search'))                                # Result : False  // expected : False
print(rp.can_fetch("*", 'https://www.google.com/search/howsearchworks'))                 # Result : False  // expected : True
print(rp.can_fetch("*", 'https://www.google.com/groups'))                                # Result : False  // expected : False
print(rp.can_fetch("*", 'https://www.google.com/citations?user=test'))                   # Result : False  // expected : True

[terryjreedy]

The robotparser has the wrong link for a disciption of the parsed format, and the updated doc is incomplete. I will try to fix soon.

I otherwise suspect that the errors reported above are duplicates of previous issues.

[ronaldoussoren]

The example robots.txt files that the OP links to contain wildcards in the paths, that's something urllib.robotparser doesn't support at the moment.

See also #114310.

[aravindkarnam]

@picnixz @ronaldoussoren What's the latest on this bug? Is this being patched?

[picnixz]

I don't know. We seem to have issues regarding the specs to choose and whether to fix/drop the support for * as there is nothing telling us exactly what we should do. If you want to research official documents for specs about robotparsers (RFCs or more recent Google guidelines, maybe Mozilla references as well), then it would be helpful.

[aravindkarnam]

@picnixz Thanks for the quick turnaround. Sure. I'll do some research around this and share my findings here.