You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| Total requests | The total number of requests to crawl your website, from all AI crawlers |
25
34
| Blocked requests | The number of requests you have blocked (by blocking AI crawlers) |
26
35
| Allowed requests | The number of requests you have allowed (by allowing AI crawlers) |
27
-
| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler |
28
-
| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler |
29
36
| Hosts | The owner of the AI crawler |
30
37
| Overall popular paths | The most popular pages crawled by AI crawlers, from all AI crawlers |
31
38
| Most active AI crawlers by operators | The AI crawler owners with the highest number of requests to access your site |
39
+
| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler |
40
+
| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler |
41
+
| Referrals | A graph which displays the number of referrals from each AI operator |
42
+
| Referers | The list of referers who directed traffic to your site |
32
43
33
44
:::note[Requests in AI Audit metrics]
34
45
The number of requests in AI Audit metrics are specifically requests which were met with HTTP code 200 (the request was successfully served, with actual content).
Copy file name to clipboardExpand all lines: src/content/docs/ai-audit/features/manage-ai-crawlers.mdx
+32-33Lines changed: 32 additions & 33 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
title: Manage AI crawlers
3
3
pcx_content_type: concept
4
4
sidebar:
5
-
order: 2
5
+
order: 5
6
6
---
7
7
8
8
import { Steps, GlossaryTooltip, Tabs, TabItem, Example } from"~/components";
@@ -15,7 +15,7 @@ To manage AI crawlers:
15
15
2. Go to **AI Audit**.
16
16
3. Go to the **AI Crawlers** tab.
17
17
18
-
## View the list of AI crawlers
18
+
## Review AI crawler activity
19
19
20
20
The **AI Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information.
21
21
@@ -47,6 +47,16 @@ On the free plan, AI Audit identifies AI crawlers based on their [user agent str
47
47
Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field.
48
48
:::
49
49
50
+
### Filter AI crawler data
51
+
52
+
You can use filters to narrow the scope of your result.
53
+
54
+
-**Name:** Search the name of the AI crawler.
55
+
-**Operator:** Filter by the AI crawler operator.
56
+
-**Category:** Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).
57
+
58
+
The values of the table will update according to your filter.
59
+
50
60
## Take action for each AI crawler
51
61
52
62
<Tabs>
@@ -64,31 +74,30 @@ For each AI crawler, you can take one of three actions: allow, charge, or block.
64
74
65
75
<Exampletitle="Allow access">
66
76
67
-
You can allow an AI crawler to scrape your content.
68
-
69
-
From the **Actions** column, select **Allow**.
70
-
77
+
-**Summary:** You can allow an AI crawler to scrape your content.
78
+
-**When to use:** Allow AI crawlers that offer services which provide value through citations, referrals, or existing agreements.
79
+
-**Implementation:** From the **Actions** column, select **Allow**.
71
80
Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
72
81
73
82
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
74
83
75
84
</Example>
76
85
77
-
<Exampletitle="Charge for crawl (Enterprise only)">
78
-
79
-
You can charge the owner of the AI crawler for each crawl request.
80
-
81
-
From the **Actions** column, select **Charge**.
82
-
83
-
For more information, refer to [What is Pay Per Crawl?](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/).
86
+
<Exampletitle="Block access">
84
87
88
+
-**Summary:** You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
89
+
-**When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.
90
+
-**Implementation:** From the **Actions** column, select **Block**.
85
91
</Example>
86
92
87
-
<Exampletitle="Block access">
93
+
<Exampletitle="Charge for crawl (private beta)">
88
94
89
-
You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
95
+
-**Summary:** You can charge the owner of the AI crawler for each crawl request.
96
+
-**When to use:** Charge AI crawlers when your content has training value, and you want to explore monetization options
97
+
-**Implementation:** From the **Actions** column, select **Charge**.
98
+
99
+
For more information, refer to [What is Pay Per Crawl?](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/).
90
100
91
-
From the **Actions** column, select **Block**.
92
101
</Example>
93
102
94
103
</TabItem>
@@ -98,31 +107,21 @@ For each AI crawler, you can choose to allow or block access.
98
107
99
108
<Exampletitle="Allow access">
100
109
101
-
You can allow an AI crawler to scrape your content. Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
102
-
103
-
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
104
-
110
+
-**Summary:**You can allow an AI crawler to scrape your content.
111
+
-**When to use:** Allow AI crawlers that offer services which provide value through citations, referrals, or existing agreements.
112
+
-**Implementation:** From the **Actions** column, select **Allow**.
113
+
Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
105
114
</Example>
106
115
107
116
<Exampletitle="Block access">
108
-
You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
109
-
110
-
Use the toggles in the **Block** column to block specific AI crawlers from accessing your content.
117
+
-**Summary:**You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
118
+
-**When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.
119
+
-**Implementation:** From the **Actions** column, select **Block**.
111
120
</Example>
112
121
113
122
</TabItem>
114
123
</Tabs>
115
124
116
-
## Filter AI crawler data
117
-
118
-
You can use filters to narrow the scope of your result.
119
-
120
-
-**Name:** Search the name of the AI crawler.
121
-
-**Operator:** Filter by the AI crawler operator.
122
-
-**Category:** Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).
123
-
124
-
The values of the table will update according to your filter.
125
-
126
125
## Related resources
127
126
128
127
- Use [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) to charge AI crawlers every time they access your content.
-**Explore monetization options** - Set up pay per crawl pricing for content access (private beta)
32
32
-**Deploy with zero configuration** - Works automatically on all Cloudflare plans
33
33
34
-
<LinkButtonhref="/ai-audit/get-started/">Get started </LinkButton>
35
-
36
34
:::note[Beta phase]
37
35
AI Audit is currently only available as a beta product.
38
36
:::
39
37
38
+
<LinkButtonhref="/ai-audit/get-started/">Get started </LinkButton>
39
+
40
40
---
41
41
42
42
## Features
@@ -50,9 +50,9 @@ AI Audit is currently only available as a beta product.
50
50
</Feature>
51
51
52
52
<Feature
53
-
header="Analyze AI crawlers"
54
-
href="/ai-audit/features/analyze-ai-crawlers/"
55
-
cta="Analyze AI crawlers"
53
+
header="Analyze AI traffic"
54
+
href="/ai-audit/features/analyze-ai-traffic/"
55
+
cta="Analyze AI traffic"
56
56
>
57
57
Gain insight into how AI crawlers are interacting with your pages.
58
58
</Feature>
@@ -67,6 +67,22 @@ AI Audit is currently only available as a beta product.
67
67
68
68
---
69
69
70
+
## Use cases
71
+
72
+
<Cardtitle="Publishers and content creators">
73
+
Publishers and content creators can monitor which AI crawlers are accessing their articles and educational content. Set policies to allow beneficial crawlers while blocking others.
74
+
</Card>
75
+
76
+
<Cardtitle="E-commerce and business sites">
77
+
E-commerce and business sites can identify AI crawler activity on product pages and business information. Control access to sensitive data like pricing and inventory.
78
+
</Card>
79
+
80
+
<Cardtitle="Documentation sites">
81
+
Documentation sites can track how AI crawlers are accessing their technical documentation. Gain insight into how AI crawlers are engaging with your site.
| AI crawler detection via user agent strings | Advanced AI crawler detection via [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields)|
9
-
| Maximum 24-hour analytics window | Maximum 30-day analytics window |
10
-
| Allow/block controls | Allow/block controls, and the ability to charge AI crawlers using [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/)|
| AI crawler detection via [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent)| Advanced AI crawler detection via [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields)|
9
+
| Maximum 24-hour analytics window | Maximum 30-day analytics window |
10
+
| Allow/block controls | Allow/block controls, and the ability to charge AI crawlers using [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/)|
0 commit comments