[3.0] Automatically populate robots.txt

[Sesquipedalian]

Provides SMF with the ability to add rules to robots.txt.

• Adds a new setting, Config::$modSettings['robots_txt'], with the path to the robots.txt file.
• To assist admins, the relevant ACP page offers to populate the setting with the correct path.
• When a valid and writable path is saved in this setting, SMF will find the file and add rules to it as necessary.
• If the path is not writable, the admin will be warned and SMF will not attempt to update the file.
• If the path is writable, SMF adds rules to tell all spiders that they should ignore URLs that match the pattern /path/to/index.php?msg= (where /path/to/index.php will be set to the appropriate value for the individual forum instance), as well as URLs containing PHPSESSID or ;topicseen.
• Redundancy checks are performed to ensure that if a rule is already present in the file, it will not be added again.

Fixes #8367

[Sesquipedalian]

@sbulen and @Oldiesmann, I'd be glad to have your thoughts on this too.

[sbulen]

I came back to provide the requested feedback & see it's been merged already...

I strongly prefer the other solutions we discussed, that would eliminate the redirects (though not all the msg links). Or the proposal to hide message-specific links from bots.

The problem with a robots.txt solution is that robots.txt is actually kinda complex, and can have different rules for different bots. I find I need to make use of that, as some crawlers honor some directives & some don't. And this small set of rules is only a tiny portion of what is needed. And I'm not sure we want to automate providing everything that is needed, because different forums are configured differently. E.g., some allow guests viewing attachments & some don't. Some allow guests viewing calendar & some don't.

A clarification on the canonical: https://myforum.net/index.php?topic=333333.msg22222 does not match canonical. For topics & messages, only https://myforum.net/index.php?topic=333333.999 matches canonical.

For this reason I have added a ".msg" disallow to my robots.txt as well.

As far as I know, there is no message-specific canonical. And I don't think there should be.