Wikimedia robot policy

[p3palazzo]

Wikimedia commons has been restricting live downloading of resources, which prevents Pandoc from retrieving images for PDF output, for example. I wonder if anything can be done to make Pandoc comply with their Robot policy. The error specifically states:

Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also https://phabricator.wikimedia.org/T400119.

[jgm]

I don't think pandoc itself counts as a "robot." If you run a script that uses pandoc to request many sites from wikipedia, so that Wikipedia is blocking, then your script is a robot and should be made to comply with the policy. For pandoc each run is separate, so in making a request to Wikipedia, it doesn't know that a previous request received a 429, for example.

[alerque]

This is all correct, but robot or not it should still set a valid user-agent string. Not setting that will get it treated as an unidentified robot by default.

[jgm]

What should it set , do you think ?

[jgm]

pandoc/VERSION presumably ?

Note that it is already possible to set this manually, as the manual says:

pandoc -f html -t markdown --request-header User-Agent:"pandoc/3.9.0.1" \
  https://www.fsf.org

[alerque]

Yes, pandoc/VERSION would be a good start. I would also consider stuffing haskell-http-client/VERSION in there since it is relevant to have the server should expect the client to behave over the wire.

You are quite right that if the user wrote a script and uses Pandoc to make a bunch of requests, their script is the robot and they should manually identify themselves as the user agent using their own --request-header. Also at that point they should be fetching robots.txt and other things up front before they get started with other things. For a one-off request from a user Pandoc doesn't need to do that directly.

[jgm]

I just pushed a commit that sets pandoc/VERSION as default user agent; this can be overridden with the option mentioned.

[p3palazzo]

Sorry I missed that in the docs! It worked as expected (though Wikimedia still complained the script makes too many requests per minute, so in the end I will have to download the files manually beforehand).