Skip to content

[AI Audit] Adding note on detection, adding note on PPC beta #23386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 2, 2025
Merged

[AI Audit] Adding note on detection, adding note on PPC beta #23386

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions src/content/docs/ai-audit/features/manage-ai-crawlers.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,25 @@ The **AI Crawlers** tab displays a table of AI crawlers that are requesting acce
</TabItem>
</Tabs>

:::note[Quality of AI crawler detection]
On the free plan, AI Audit identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Audit to detect easy-to-detect (well-known) AI crawlers.

Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field.
:::

## Take action for each AI crawler

<Tabs>
<TabItem label="With pay per crawl">

:::note[Pay per crawl closed beta]
Pay per crawl is currently in closed beta.

To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer.

To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/).
:::

For each AI crawler, you can take one of three actions: allow, charge, or block.

<Example title="Allow access">
Expand Down