AI Crawl Control: Reflect Dashboard Changes and Update Pay Per Crawl Site Owner Onboarding

[CameronWhiteside]

Summary

This PR updates the AI Crawl Control documentation to reflect recent dashboard changes and significantly improves the Pay Per Crawl site owner onboarding experience. The changes streamline the user journey, update terminology, and provide clearer guidance for site owners looking to monetize their content through AI crawler access.

Small Changes

• Updated "AI Crawlers" tab to "Crawlers" across all documentation to reflect UI
• Updated "Blocked requests" to "Unsuccessful requests" to better reflect the actual behavior
• Moved FAQ to pay per crawl since content does not apply to AI Crawl Control
• Add clear warnings about blocking search engine crawlers
• Add bulk control instructions for managing multiple crawlers efficiently
• Add references to WAF rules for custom configurations

Pay Per Crawl Site Owner Workflow Restructuring

• Replaced complex Stripe-first workflow with account-level configuration approach
• New workflow order:
  1. Enable in account settings
  2. Set pay per crawl price
  3. Select crawlers to charge
  4. Monitor activity
  5. Manage payouts

Files Changed

New Files Added

• src/content/docs/ai-crawl-control/features/pay-per-crawl/faq.mdx - Comprehensive FAQ for both site owners and AI bot operators
• src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings.mdx - Account-level configuration guide
• src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts.mdx - Stripe integration and payout management
• src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity.mdx - Activity monitoring and optimization guidance

Files Removed

• connect-to-stripe.mdx - Stripe connection moved to later in workflow
• set-up-cloudflare-account.mdx - Basic account setup removed (assumed prerequisite, noted in step 1)
• monitor-metrics.mdx - Consolidated into broader activity monitoring for pay per crawl

Note: This PR reflects dashboard changes that have already been implemented in the product. The documentation updates ensure users have accurate, up-to-date guidance for using AI Crawl Control and Pay Per Crawl features.

Documentation checklist

• The documentation style guide has been adhered to.
• If a larger change - such as adding a new page- an issue has been opened in relation to any incorrect or out of date information that this PR fixes.
• Files which have changed name or location have been allocated redirects.