diff --git a/src/content/docs/ai-audit/features/manage-ai-crawlers.mdx b/src/content/docs/ai-audit/features/manage-ai-crawlers.mdx index 0f94f1897d3d19..2b776d64a8e6b7 100644 --- a/src/content/docs/ai-audit/features/manage-ai-crawlers.mdx +++ b/src/content/docs/ai-audit/features/manage-ai-crawlers.mdx @@ -41,11 +41,25 @@ The **AI Crawlers** tab displays a table of AI crawlers that are requesting acce +:::note[Quality of AI crawler detection] +On the free plan, AI Audit identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Audit to detect easy-to-detect (well-known) AI crawlers. + +Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field. +::: + ## Take action for each AI crawler +:::note[Pay per crawl closed beta] +Pay per crawl is currently in closed beta. + +To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer. + +To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/). +::: + For each AI crawler, you can take one of three actions: allow, charge, or block.