Modify robots.txt for AI training and access

[Mr-Autistic]

Updated robots.txt configurations to allow AI training for specific user agents and modified disallow rules.

Summary

Screenshots (optional)

Documentation checklist

• Is there a changelog entry (guidelines)? If you don't add one for something awesome and new (however small) — how will our customers find out? Changelogs are automatically posted to RSS feeds, the Discord, and X.
• The change adheres to the documentation style guide.
• If a larger change - such as adding a new page- an issue has been opened in relation to any incorrect or out of date information that this PR fixes.
• Files which have changed name or location have been allocated redirects.

[Mr-Autistic]

thank you