Skip to content

Commit 2d21d0f

Browse files
committed
Edits
1 parent 3076bb8 commit 2d21d0f

File tree

5 files changed

+11
-8
lines changed

5 files changed

+11
-8
lines changed

src/content/changelog/ai-audit/2025-07-01-pay-per-crawl.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
title: Introducing pay per crawl private beta
33
description: A new feature which allows site owners to charge crawlers for access to their content
4-
date: 2025-07-01T11:00:00Z
4+
date: 2025-07-01T12:00:00Z
55
---
66

77
We are introducing a new feature of AI Audit - pay per crawl. Pay per crawl enables site owners to require payment from AI crawlers every time the crawlers access their content, thereby fostering a fairer Internet by enabling site owners to control and monetize how their content gets used by AI.
File renamed without changes.

src/content/docs/ai-audit/features/manage-ai-crawlers.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ You can use filters to narrow the scope of your result.
9999
- **Operator:** Filter by the AI crawler operator.
100100
- **Category:** Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).
101101

102-
The values of the AI Audit metrics will update according to your filter.
102+
The values of the table will update according to your filter.
103103

104104
## Related resources
105105

src/content/docs/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl.mdx

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,14 @@ title: What is Pay Per Crawl?
33
pcx_content_type: concept
44
sidebar:
55
order: 2
6-
hidden: true
76
---
87

98
import { Steps, GlossaryTooltip } from "~/components";
109

1110
:::note[Closed beta]
1211
Pay per crawl is currently in closed beta.
1312

14-
To find out how to join the beta program and to learn more about pay per crawl, refer to [Blog].
13+
To find out how to join the beta program and to learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/).
1514
:::
1615

1716
AI crawlers often consume vast amounts of web content. Some provide mutual benefit to content owners by indexing content for search engines, but others engage in activities such as content scraping without permission.
@@ -41,5 +40,5 @@ Refer to the following resources.
4140

4241
- [Use pay per crawl as a site owner](/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account/).
4342
- [Use pay per crawl as an AI owner](/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/set-up-cloudflare-account/).
44-
- [Pay per crawl with Cloudflare WAF](/ai-audit/configuration/ai-audit-with-waf/).
45-
- [Pay per crawl with Cloudflare Bots](/ai-audit/configuration/ai-audit-with-bots/).
43+
- [AI Audit with Cloudflare WAF](/ai-audit/configuration/ai-audit-with-waf/).
44+
- [AI Audit with Cloudflare Bots](/ai-audit/configuration/ai-audit-with-bots/).

src/content/docs/ai-audit/get-started.mdx

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ import { Details, Render, Steps, Tabs, TabItem, GlossaryTooltip } from "~/compon
1616
This guide instructs you on how to:
1717

1818
- View AI crawlers that are interacting with pages in your domain (a [Cloudflare zone](/fundamentals/concepts/accounts-and-zones/#zones)).
19-
- Create a rule to block AI crawlers on your pages.
19+
- Use AI Audit to block individual crawlers from accessing your content.
2020

2121
## Prerequisites
2222

@@ -45,7 +45,7 @@ On the free plan, AI Audit identifies AI crawlers based on their [user agent str
4545
Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field.
4646
:::
4747

48-
</TabItem> <TabItem label="Paid plan">
48+
</TabItem> <TabItem label="Paid plans">
4949

5050
To only block specific AI crawlers:
5151

@@ -63,6 +63,10 @@ To only block specific AI crawlers:
6363
</TabItem>
6464
</Tabs>
6565

66+
For more information, refer to [Manage AI crawlers](/ai-audit/features/manage-ai-crawlers/).
67+
68+
You can also create more complex rules when taking action on AI crawlers, using [Cloudflare WAF](/waf/). For more information on creating more specific rules, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
69+
6670
## 2. Review detected AI crawlers
6771

6872
<Tabs>

0 commit comments

Comments
 (0)