You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| Total requests | The total number of requests to crawl your website, from all AI crawlers |
25
34
| Blocked requests | The number of requests you have blocked (by blocking AI crawlers) |
26
35
| Allowed requests | The number of requests you have allowed (by allowing AI crawlers) |
27
-
| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler |
28
-
| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler |
29
36
| Hosts | The owner of the AI crawler |
30
37
| Overall popular paths | The most popular pages crawled by AI crawlers, from all AI crawlers |
31
38
| Most active AI crawlers by operators | The AI crawler owners with the highest number of requests to access your site |
39
+
| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler |
40
+
| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler |
41
+
| Referrals | A graph which displays the number of referrals from each AI operator |
42
+
| Referers | The list of referers who directed traffic to your site |
32
43
33
44
:::note[Requests in AI Audit metrics]
34
45
The number of requests in AI Audit metrics are specifically requests which were met with HTTP code 200 (the request was successfully served, with actual content).
Copy file name to clipboardExpand all lines: src/content/docs/ai-audit/features/manage-ai-crawlers.mdx
+32-33Lines changed: 32 additions & 33 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
title: Manage AI crawlers
3
3
pcx_content_type: concept
4
4
sidebar:
5
-
order: 2
5
+
order: 5
6
6
---
7
7
8
8
import { Steps, GlossaryTooltip, Tabs, TabItem, Example } from"~/components";
@@ -15,7 +15,7 @@ To manage AI crawlers:
15
15
2. Go to **AI Audit**.
16
16
3. Go to the **AI Crawlers** tab.
17
17
18
-
## View the list of AI crawlers
18
+
## Review AI crawler activity
19
19
20
20
The **AI Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information.
21
21
@@ -47,6 +47,16 @@ On the free plan, AI Audit identifies AI crawlers based on their [user agent str
47
47
Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field.
48
48
:::
49
49
50
+
### Filter AI crawler data
51
+
52
+
You can use filters to narrow the scope of your result:
53
+
54
+
-**Name:** Search the name of the AI crawler.
55
+
-**Operator:** Filter by the AI crawler operator.
56
+
-**Category:** Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).
57
+
58
+
The values of the table will update according to your filter.
59
+
50
60
## Take action for each AI crawler
51
61
52
62
<Tabs>
@@ -64,31 +74,30 @@ For each AI crawler, you can take one of three actions: allow, charge, or block.
64
74
65
75
<Exampletitle="Allow access">
66
76
67
-
You can allow an AI crawler to scrape your content.
68
-
69
-
From the **Actions** column, select **Allow**.
70
-
77
+
-**Summary:** You can allow an AI crawler to scrape your content.
78
+
-**When to use:** Allow AI crawlers that offer services which provide value through citations, referrals, or existing agreements.
79
+
-**Implementation:** From the **Actions** column, select **Allow**.
71
80
Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
72
81
73
82
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
74
83
75
84
</Example>
76
85
77
-
<Exampletitle="Charge for crawl (Enterprise only)">
78
-
79
-
You can charge the owner of the AI crawler for each crawl request.
80
-
81
-
From the **Actions** column, select **Charge**.
82
-
83
-
For more information, refer to [What is Pay Per Crawl?](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/).
86
+
<Exampletitle="Block access">
84
87
88
+
-**Summary:** You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
89
+
-**When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.
90
+
-**Implementation:** From the **Actions** column, select **Block**.
85
91
</Example>
86
92
87
-
<Exampletitle="Block access">
93
+
<Exampletitle="Charge for crawl (private beta)">
88
94
89
-
You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
95
+
-**Summary:** You can charge the owner of the AI crawler for each crawl request.
96
+
-**When to use:** Charge AI crawlers when your content has training value, and you want to explore monetization options
97
+
-**Implementation:** From the **Actions** column, select **Charge**.
98
+
99
+
For more information, refer to [What is Pay Per Crawl?](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/).
90
100
91
-
From the **Actions** column, select **Block**.
92
101
</Example>
93
102
94
103
</TabItem>
@@ -98,31 +107,21 @@ For each AI crawler, you can choose to allow or block access.
98
107
99
108
<Exampletitle="Allow access">
100
109
101
-
You can allow an AI crawler to scrape your content. Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
102
-
103
-
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
104
-
110
+
-**Summary:**You can allow an AI crawler to scrape your content.
111
+
-**When to use:** Allow AI crawlers that offer services which provide value through citations, referrals, or existing agreements.
112
+
-**Implementation:** From the **Actions** column, select **Allow**.
113
+
Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
105
114
</Example>
106
115
107
116
<Exampletitle="Block access">
108
-
You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
109
-
110
-
Use the toggles in the **Block** column to block specific AI crawlers from accessing your content.
117
+
-**Summary:**You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
118
+
-**When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.
119
+
-**Implementation:** From the **Actions** column, select **Block**.
111
120
</Example>
112
121
113
122
</TabItem>
114
123
</Tabs>
115
124
116
-
## Filter AI crawler data
117
-
118
-
You can use filters to narrow the scope of your result.
119
-
120
-
-**Name:** Search the name of the AI crawler.
121
-
-**Operator:** Filter by the AI crawler operator.
122
-
-**Category:** Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).
123
-
124
-
The values of the table will update according to your filter.
125
-
126
125
## Related resources
127
126
128
127
- Use [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) to charge AI crawlers every time they access your content.
Copy file name to clipboardExpand all lines: src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-metrics.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,7 +21,7 @@ Once you have confirmed the selection of AI crawlers to charge, monitor key metr
21
21
{/* prettier-ignore */}
22
22
<Steps>
23
23
1. Go to the **Metrics** tab.
24
-
2.[View AI Audit metrics](/ai-audit/features/analyze-ai-crawlers/#view-ai-audit-metrics). Understanding these metrics can help you better manage AI crawlers accessing your content.
24
+
2.[View AI Audit metrics](/ai-audit/features/analyze-ai-traffic/#view-ai-audit-metrics). Understanding these metrics can help you better manage AI crawlers accessing your content.
Analyze and control how third-party AI crawlers access and interact with your website.
20
+
Monitor and control how AI services access your website content.
18
21
19
22
</Description>
20
23
21
-
<Plantype="all" />
24
+
AI companies are increasingly using web content to train their models and power AI applications. AI Audit gives you visibility into which AI services are accessing your content, and provides tools to manage access according to your preferences.
22
25
23
-
AI Audit helps manage AI crawlers on your website by providing visibility on which crawlers are accessing your pages, and allowing you to block unwanted crawlers.
26
+
With AI Audit, you can:
24
27
25
-
<LinkButtonhref="/ai-audit/get-started/">Get started </LinkButton>
28
+
-**See which AI services access your content** - Monitor the dashboard to see crawler activity and request patterns
29
+
-**Control access with granular policies** - Set allow or block rules for individual crawlers
30
+
-**Monitor robots.txt compliance** - Track which crawlers follow your directives and create enforcement rules
31
+
-**Explore monetization options** - Set up pay per crawl pricing for content access [(private beta)](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/)
32
+
-**Deploy with zero configuration** - Works automatically on all Cloudflare plans
26
33
27
34
:::note[Beta phase]
28
35
AI Audit is currently only available as a beta product.
29
36
:::
30
37
38
+
<LinkButtonhref="/ai-audit/get-started/">Get started </LinkButton>
39
+
31
40
---
32
41
33
42
## Features
@@ -41,9 +50,9 @@ AI Audit is currently only available as a beta product.
41
50
</Feature>
42
51
43
52
<Feature
44
-
header="Analyze AI crawlers"
45
-
href="/ai-audit/features/analyze-ai-crawlers/"
46
-
cta="Analyze AI crawlers"
53
+
header="Analyze AI traffic"
54
+
href="/ai-audit/features/analyze-ai-traffic/"
55
+
cta="Analyze AI traffic"
47
56
>
48
57
Gain insight into how AI crawlers are interacting with your pages.
49
58
</Feature>
@@ -58,6 +67,22 @@ AI Audit is currently only available as a beta product.
58
67
59
68
---
60
69
70
+
## Use cases
71
+
72
+
<Cardtitle="Publishers and content creators">
73
+
Publishers and content creators can monitor which AI crawlers are accessing their articles and educational content. Set policies to allow beneficial crawlers while blocking others.
74
+
</Card>
75
+
76
+
<Cardtitle="E-commerce and business sites">
77
+
E-commerce and business sites can identify AI crawler activity on product pages and business information. Control access to sensitive data like pricing and inventory.
78
+
</Card>
79
+
80
+
<Cardtitle="Documentation sites">
81
+
Documentation sites can track how AI crawlers are accessing their technical documentation. Gain insight into how AI crawlers are engaging with your site.
82
+
</Card>
83
+
84
+
---
85
+
61
86
## Related Products
62
87
63
88
<RelatedProduct
@@ -75,3 +100,11 @@ Identify and mitigate automated traffic to protect your domain from bad bots.
75
100
>
76
101
Get automatic protection from vulnerabilities and the flexibility to create custom rules.
| AI crawler detection via [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent)| Advanced AI crawler detection via [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields)|
9
+
| Maximum 24-hour analytics window | Configurable analytics timeframes |
10
+
| Allow/block controls | Allow/block controls, and the ability to charge AI crawlers using [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/)|
0 commit comments