Skip to content

Commit 4c5ceb3

Browse files
CameronWhitesideCameronWhiteside
committed
AI Crawl Control: updated to reflect dash changes and pay per crawl onboarding
AI Crawl Control: removed duplicate FAQ AI Crawl Control: fixed missing links AI Crawl Control: fixed missing links
1 parent 3e9068f commit 4c5ceb3

File tree

17 files changed

+332
-238
lines changed

17 files changed

+332
-238
lines changed

‎src/content/docs/ai-crawl-control/features/analyze-ai-traffic.mdx‎

Lines changed: 14 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -16,38 +16,31 @@ To view AI Crawl Control metrics:
1616
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
1717
2. Go to **AI Crawl Control**.
1818

19-
You can find meaningful information across both **AI Crawlers** and **Metrics** tabs.
19+
You can find meaningful information across both **Crawlers** and **Metrics** tabs.
2020

2121
## View AI crawler activity
2222

23-
The **AI Crawlers** tab provides you with the following information:
23+
The **Crawlers** tab provides you with the following information:
2424

25-
- Complete list of detected AI crawlers and their operators
25+
- Total requests to crawl your website from common AI crawlers
2626
- Number of requests made by each AI crawler
2727
- Number of `robots.txt` violations for each crawler
2828

2929
## View AI Crawl Control metrics
3030

3131
The **Metrics** tab provides you with the following metrics to help you understand how AI crawlers are interacting with your website.
3232

33-
| Metric | Description |
34-
| ------------------------------------ | ----------------------------------------------------------------------------- |
35-
| Total requests | The total number of requests to crawl your website, from all AI crawlers |
36-
| Blocked requests | The number of crawler requests you have blocked, from any rule |
37-
| Allowed requests | The number of crawler requests you have allowed |
38-
| Hosts | The owner of the AI crawler |
39-
| Overall popular paths | The most popular pages crawled by AI crawlers, from all AI crawlers |
40-
| Most active AI crawlers by operators | The AI crawler owners with the highest number of requests to access your site |
41-
| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler |
42-
| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler |
43-
| Referrals | A graph which displays the number of referrals from each AI operator |
44-
| Referers | The list of referers who directed traffic to your site |
45-
46-
:::note[Requests in AI Crawl Control metrics]
47-
The number of requests in AI Crawl Control metrics are specifically requests which were met with HTTP code 200 (the request was successfully served, with actual content).
48-
49-
AI Crawl Control metrics filter all other HTTP codes.
50-
:::
33+
| Metric | Description |
34+
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
35+
| Total requests | The total number of requests to crawl your website, from all AI crawlers |
36+
| Allowed requests | The number of crawler requests that received a successful response from your site |
37+
| Unsuccessful requests | The number of crawler requests that failed (HTTP 4xx or 5xx) as a result of a blocked request, other security rules, or website errors such as a crawler attempting to access a non-existent page |
38+
| Overall popular paths | The most popular pages crawled by AI crawlers, from all AI crawlers |
39+
| Most active AI crawlers by operators | The AI crawler owners with the highest number of requests to access your site |
40+
| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler |
41+
| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler |
42+
| Referrals | A graph which displays the number of visits that were directed to your site from each AI operator |
43+
| Referers | The list of referers who directed visits to your site |
5144

5245
## Filter date range
5346

‎src/content/docs/ai-crawl-control/features/manage-ai-crawlers.mdx‎

Lines changed: 46 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,22 @@ sidebar:
55
order: 5
66
---
77

8-
import { Steps, GlossaryTooltip, Tabs, TabItem, Example, Plan } from "~/components";
8+
import {
9+
Steps,
10+
GlossaryTooltip,
11+
Tabs,
12+
TabItem,
13+
Example,
14+
Plan,
15+
} from "~/components";
916

1017
AI Crawl Control enables you to take specific action for each AI crawler.
1118

1219
To manage AI crawlers:
1320

1421
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
1522
2. Go to **AI Crawl Control**.
16-
3. Go to the **AI Crawlers** tab.
23+
3. Go to the **Crawlers** tab.
1724

1825
## Review AI crawler activity
1926

@@ -23,12 +30,12 @@ The **Crawlers** tab displays a table of AI crawlers that are requesting access
2330
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
2431
| Crawler | The name of the AI crawler and the operator that owns it. |
2532
| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/#categories). |
26-
| Requests | Total allowed and blocked requests with trend chart. Blocked requests may come from any configured rule, not just the actions shown here. |
33+
| Requests | Total allowed and unsuccessful requests with trend chart. Unsuccessful requests may come from any rule or response error, not just the block action in AI Crawl Control. |
2734
| Robots.txt violations | The number of times the AI crawler has violated your <GlossaryTooltip term="robots.txt">`robots.txt`</GlossaryTooltip> file. |
2835
| Action | The action you wish to take for the AI crawler. Refer to [Take action for each AI crawler](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). |
2936

3037
:::note[Quality of AI crawler detection]
31-
On the free plan, AI Crawl Control identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Crawl Control to detect easy-to-detect (well-known) AI crawlers.
38+
On the free plan, AI Crawl Control identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Crawl Control to detect well-known, self-identifying AI crawlers.
3239

3340
Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field.
3441
:::
@@ -46,31 +53,21 @@ The values of the table will update according to your filter.
4653
## Take action for each AI crawler
4754

4855
<Tabs>
49-
<TabItem label="With pay per crawl">
50-
51-
:::note[Pay per crawl closed beta]
52-
Pay per crawl is currently in closed beta.
53-
54-
To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer.
55-
56-
To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/).
57-
:::
56+
<TabItem label="Without pay per crawl">
5857

59-
For each AI crawler, you can take one of three actions: allow, charge, or block.
58+
For each AI crawler, you can choose to allow or block access.
6059

6160
<Example title="Allow access">
6261

6362
- **Summary:** You can allow an AI crawler to scrape your content.
6463
- **When to use:** Allow AI crawlers that offer services which provide value through citations, referrals, or existing agreements.
6564
- **Implementation:** From the **Actions** column, select **Allow**.
66-
Note that you can still choose to [Enforce `robots.txt`](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
6765

68-
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
66+
Note that you can still choose to [Enforce `robots.txt`](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
6967

7068
</Example>
7169

7270
<Example title="Block access">
73-
7471
- **Summary:** You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
7572
- **When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.
7673
- **Implementation:** From the **Actions** column, select **Block**.
@@ -79,32 +76,32 @@ Note that you can configure the response that gets returned when blocking an AI
7976

8077
</Example>
8178

82-
<Example title="Charge for crawl (private beta)">
83-
84-
- **Summary:** You can charge the owner of the AI crawler for each crawl request.
85-
- **When to use:** Charge AI crawlers when your content has training value, and you want to explore monetization options
86-
- **Implementation:** From the **Actions** column, select **Charge**.
79+
</TabItem>
80+
<TabItem label="With pay per crawl">
8781

88-
For more information, refer to [What is Pay Per Crawl?](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/).
82+
:::note[Pay per crawl closed beta]
83+
Pay per crawl is currently in closed beta.
8984

90-
</Example>
85+
To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer.
9186

92-
</TabItem>
93-
<TabItem label="Without pay per crawl">
87+
To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/).
88+
:::
9489

95-
For each AI crawler, you can choose to allow or block access.
90+
For each AI crawler, you can take one of three actions: allow, charge, or block.
9691

9792
<Example title="Allow access">
9893

9994
- **Summary:** You can allow an AI crawler to scrape your content.
10095
- **When to use:** Allow AI crawlers that offer services which provide value through citations, referrals, or existing agreements.
10196
- **Implementation:** From the **Actions** column, select **Allow**.
97+
Note that you can still choose to [Enforce `robots.txt`](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
10298

103-
Note that you can still choose to [Enforce `robots.txt`](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
99+
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
104100

105101
</Example>
106102

107103
<Example title="Block access">
104+
108105
- **Summary:** You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
109106
- **When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.
110107
- **Implementation:** From the **Actions** column, select **Block**.
@@ -113,9 +110,24 @@ Note that you can configure the response that gets returned when blocking an AI
113110

114111
</Example>
115112

113+
<Example title="Charge for crawl (private beta)">
114+
115+
- **Summary:** You can charge the owner of the AI crawler for each crawl request.
116+
- **When to use:** Charge AI crawlers when your content has training value, and you want to explore monetization options.
117+
- **Implementation:** From the **Actions** column, select **Charge**.
118+
119+
For more information, refer to [What is Pay Per Crawl?](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/).
120+
121+
</Example>
122+
116123
</TabItem>
124+
117125
</Tabs>
118126

127+
:::tip[Need more advanced control?]
128+
You can also create more complex rules when taking action on AI crawlers, using [Cloudflare WAF](/waf/). For more information on creating more specific rules, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
129+
:::
130+
119131
## Configure block response
120132

121133
<Plan type="paid" />
@@ -130,11 +142,10 @@ This provides you with a channel to open dialogue with the AI crawler owner, and
130142
To edit these values:
131143

132144
<Steps>
133-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
134-
2. Go to **AI Crawl Control**.
135-
3. Go to the **Settings** tab.
136-
4. Under **Block response**, select **Edit**.
137-
5. Once you have edited the values, select **Save**.
145+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and
146+
select your account and domain. 2. Go to **AI Crawl Control**. 3. Go to the
147+
**Settings** tab. 4. Under **Block response**, select **Edit**. 5. Once you
148+
have edited the values, select **Save**.
138149
</Steps>
139150

140151
:::note
@@ -147,8 +158,8 @@ You can choose which HTTP response code to return when blocking an AI crawler.
147158

148159
Use the dropdown menu to select the desired response code. You can choose from:
149160

150-
- `403 Forbidden`: Use this option if you wish to indicate that you do not want the AI crawler to access your content.
151-
- `402 Payment Required`: Use this option if you wish to indicate that the AI crawler must pay to access your content.
161+
- `403 Forbidden`: Use this option if you wish to indicate that you do not want the AI crawler to access your content.
162+
- `402 Payment Required`: Use this option if you wish to indicate that the AI crawler must pay to access your content.
152163

153164
:::note
154165
Behind the scenes, AI Crawl Control uses [Cloudflare WAF](/waf/) to return custom block responses.

‎src/content/docs/ai-crawl-control/features/pay-per-crawl/faq.mdx‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
---
2+
title: Pay Per Crawl FAQ
3+
pcx_content_type: concept
4+
sidebar:
5+
order: 4
6+
---
7+
8+
import { Steps } from "~/components";
9+
10+
## Frequently asked questions for site owners
11+
12+
### Can I set different prices for different AI crawlers?
13+
14+
No. Pay per crawl allows you to configure different actions (Block, Charge, or Allow) for each crawler, but you can only set a single price that applies to all crawlers configured with the "Charge" option.
15+
16+
## Frequently asked questions for AI bot operators
17+
18+
### Will I be charged for re-crawling the same page?
19+
20+
Yes. Every time your AI crawler accesses content on a website protected with pay per crawl, it will incur the cost set by the site owner. You should implement mechanisms within your crawler to track expenditure and enforce any spending limits you want to set.
21+
22+
### Am I charged for error responses?
23+
24+
No. Charging events are only triggered for successful HTTP response codes. Error responses are not billed, even if you have sent the `crawler-exact-price` or `crawler-max-price` headers.
25+
26+
### What user agent should I use?
27+
28+
Use the standard user agents associated with your AI crawler that you have onboarded to Cloudflare and identified through Web Bot Auth.

‎src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,4 +85,4 @@ Cloudflare strongly recommends tracking and saving these values to keep an accur
8585

8686
You may wish to refer to the following resources.
8787

88-
- [AI Crawl Control FAQs](/ai-crawl-control/reference/faq/).
88+
- [Pay Per Crawl FAQs](/ai-crawl-control/features/pay-per-crawl/faq).

‎src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/index.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ pcx_content_type: navigation
44
sidebar:
55
group:
66
hideIndex: true
7-
order: 5
7+
order: 3
88
---
99

1010
import { DirectoryListing } from "~/components";

‎src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/connect-to-stripe.mdx‎

Lines changed: 0 additions & 41 deletions
This file was deleted.

‎src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings.mdx‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
title: Enable in account settings
3+
pcx_content_type: how_to
4+
sidebar:
5+
order: 2
6+
---
7+
8+
import { Steps } from "~/components";
9+
10+
```mermaid
11+
graph LR
12+
A[Enable in<br>account settings]:::highlight --> B[Set a pay per <br/>crawl price ]
13+
B --> C[Select crawlers<br>to charge]
14+
C --> D[Monitor<br>activity]
15+
D --> E[Manage<br>payouts]
16+
classDef highlight fill:#F6821F,color:white
17+
18+
click B "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price/"
19+
click C "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge/"
20+
click D "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity/"
21+
click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/"
22+
```
23+
24+
## Prerequisites
25+
26+
To configure pay per crawl, you must have the following:
27+
28+
- **Cloudflare account** - You need an active Cloudflare account with domains added
29+
- **Domain on Cloudflare** - Your domain must be using Cloudflare's nameservers or have DNS records managed by Cloudflare
30+
- **Administrator access** - Account-level configuration requires Administrator or Super Administrator permissions
31+
32+
## Configure domain access
33+
34+
An Administrator or Super Administrator with access to all domains must select which domains should show the pay per crawl controls:
35+
36+
{/* prettier-ignore */}
37+
<Steps>
38+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account.
39+
2. Go to **Manage Account** > **Settings** > **Pay Per Crawl**.
40+
3. In the **Domain Access** table, select which domains will have pay per crawl configurations visible.
41+
4. Set the **Visibility** to **Visible** for each domain you want to configure.
42+
</Steps>
43+
44+
:::note[Visibility vs. security]
45+
Setting a domain to **Visible** will not affect security rules. This only makes the pay per crawl configuration controls accessible to domain-level administrators.
46+
:::
47+
48+
After completing these steps, domain administrators can set a pay per crawl price and enable pay per crawl for their specific domains.

0 commit comments

Comments
 (0)