Issue-167: Make it possible to add values to robots.txt via the admin

[anubisthejackle]

Summary

Fixes #167

This pull request adds the ability for administrators to manage and customize the site's robots.txt output directly from the WordPress admin interface. The update introduces both site-level and network-level (for multisite installations) settings for robots.txt, allowing custom directives to be added without requiring file system access. The plugin leverages the robots_txt filter to append or prepend custom content, and all logic is encapsulated to ensure compatibility with other plugins and WordPress core.

Context

Previously, modifying robots.txt required direct file access or code changes, which was not possible in certain hosting environments (e.g., VIP, Pantheon). This update empowers both site and network administrators to manage robots.txt rules through the admin UI, without interfering with existing robots_txt filter usage from the core or other plugins.

Features

• Robots.txt Settings Section:
  A new "Robots.txt" section is available in the SEO settings page for single sites and on the network settings page for multisite installations.
• Live Preview:
  Admins can view a live preview of the compiled robots.txt output before saving changes.
• Custom Entry Fields:
  Separate fields for adding custom directives at both the site and network level.
• Multisite Support:
  Network administrators can set global rules, which are combined with individual site rules for the final output.
• Filter Implementation:
  Uses the robots_txt filter to dynamically update the robots.txt output based on saved settings.
• Refactored & Commented Code:
  Includes code clean-up, consolidated meta box logic, improved maintainability, and updated hardcoded values for consistency.

Use Case

Admins can now quickly adjust robots.txt content from the WordPress dashboard, streamlining SEO management and compliance with search engine guidelines—no code or file uploads required.

Checklist

• I have tested these changes locally.
• I have updated the documentation as needed.
• I have added tests that prove my fix is effective or that my feature works.
• I have assigned reviewers to this pull request.

[kevinfodness]

I don't want this to be managed through network options. I want this to be a per-site setting. It needs to work on both single sites and multisite, and for multisite, we should be able to configure the settings on a per-site basis.