Add feature to block AI data scraping bots

[dkotter]

Description of the Change

This PR adds a new General Settings panel under the ClassifAI Registration section that currently has one setting: Block AI Bots.

If this is turned on, we modify the robots.txt file to block the following bots:

• Applebot-Extended
• CCBot
• ClaudeBot
• FacebookBot
• Google-Extended
• GPTbot
• Meta-ExternalAgent

In doing some research, these seem to be the most widely used AI data scraping bots, though we can definitely modify this list if needed. I purposely am not blocking any AI search or assistant bots, so even if this setting is turned on, your site can still show in AI search results or AI chat results (like ChatGPT). May be worth a different setting to block those though I imagine most sites would not want those blocked.

Closes #736

How to test the Change

1. Run npm run build
2. Go to your robots.txt file and ensure you see no blocks being blocked
3. Go to the ClassifAI settings page
4. Click on ClassifAI Registration (may be worth updating this to something else now, General Settings maybe?)
5. Find the new Block AI Bots setting and toggle it on then save settings
6. Go to your robots.txt file and ensure you see bots being blocked
7. Turn the setting off, go back to the robots.txt file and ensure bots are no longer blocked

Changelog Entry

Added - New setting that when turned on, will modify your site's robots.txt file to block the most common AI data scraping bots.
Fixed - Ensure error message shows properly if registration settings are initially saved with empty values.

Credits

Props @dkotter, @jeffpaul

Checklist:

• I agree to follow this project's Code of Conduct.
• I have updated the documentation accordingly.
• I have added Critical Flows, Test Cases, and/or End-to-End Tests to cover my change.
• All new and existing tests pass.

[iamdharmesh]

Thanks for working on this @dkotter. Looks good to me

Click on ClassifAI Registration (may be worth updating this to something else now, General Settings maybe?)

Yes. As we already have a title at panel level, I think we are good to update this.

[dkotter]

Thanks for working on this @dkotter. Looks good to me

Click on ClassifAI Registration (may be worth updating this to something else now, General Settings maybe?)

Yes. As we already have a title at panel level, I think we are good to update this.

@jeffpaul Curious for your thoughts on this? In introducing this new AI block setting, we now have more than just registration settings on this page. I'm fine to leave the header ClassifAI Registration if we want but thinking something like Settings or General Settings may be better? Though also worried about confusing someone, making them think that is the page to configure all settings

[dkotter]

Note I've gone ahead and changed that menu item to be just Settings now instead of ClassifAI Registration. Happy to modify this if needed but I think this is makes more sense now that we've added this new setting