Summary

It should be possible, using this plugin, to add entries to robots.txt.

Context

We are frequently asked by clients to make adjustment's to a site's robots.txt file.

The SEO plugins we most commonly install, from Yoast and Rank Math, offer this feature, but they require permission to write to the file system, which isn't available on VIP or Pantheon.

I suggest we modify this plugin to make it possible to add to the output of the robots.txt file via the robots_txt filter.

As I imagine it, this plugin would allow adding to the output but not modifying or removing any output that comes directly from codebase (for example, a sitemap plugin adding paths to its sitemaps or core's rules to disallow crawling the admin).

A settings screen for saving the additional output would be available to users who can manage_options. On multisite networks, a network settings screen for saving additional output would be available for users who can manage_network_options. The network setting would complement the site setting, not replace it.

Use Case

It should be possible for admins to make changes to robots.txt without having to push a code update.