You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
AI Crawl Control: added docs and changelog for new crawler drilldown … (#26260)
* AI Crawl Control: added docs and changelog for new crawler drilldown features
* PCX review
* Update src/content/changelog/ai-crawl-control/2025-11-03-ai-crawl-control-crawler-info.mdx
* Update src/content/docs/ai-crawl-control/features/analyze-ai-traffic.mdx
Co-authored-by: Jun Lee <[email protected]>
* AI Crawl Control: updated changelog date
---------
Co-authored-by: Cameron Whiteside <[email protected]>
Co-authored-by: Jun Lee <[email protected]>
title: Crawler drilldowns with extended actions menu
3
+
description: Drill down into individual crawler behavior with status code charts and export data for use in WAF rules, Redirect Rules, and robots.txt files.
4
+
date: 2025-11-10
5
+
---
6
+
7
+
AI Crawl Control now supports per-crawler drilldowns with an extended actions menu and status code analytics. Drill down into Metrics, Cloudflare Radar, and Security Analytics, or export crawler data for use in [WAF custom rules](/waf/custom-rules/), [Redirect Rules](/rules/url-forwarding/), and robots.txt files.
8
+
9
+
## What's new
10
+
11
+
### Status code distribution chart
12
+
13
+
The **Metrics** tab includes a status code distribution chart showing HTTP response codes (2xx, 3xx, 4xx, 5xx) over time. Filter by individual crawler, category, operator, or time range to analyze how specific crawlers interact with your site.
14
+
15
+
16
+
17
+
### Extended actions menu
18
+
19
+
Each crawler row includes a three-dot menu with per-crawler actions:
20
+
21
+
-**View Metrics** — Filter the AI Crawl Control Metrics page to the selected crawler.
22
+
-**View on Cloudflare Radar** — Access verified crawler details on Cloudflare Radar.
23
+
-**Copy User Agent** — Copy user agent strings for use in WAF custom rules, Redirect Rules, or robots.txt files.
24
+
-**View in Security Analytics** — Filter Security Analytics by detection IDs (Bot Management customers).
25
+
-**Copy Detection ID** — Copy detection IDs for use in WAF custom rules (Bot Management customers).
26
+
27
+
28
+
29
+
## Get started
30
+
31
+
1. Log in to the Cloudflare dashboard, and select your account and domain.
32
+
2. Go to **AI Crawl Control** > **Metrics** to access the status code distribution chart.
33
+
3. Go to **AI Crawl Control** > **Crawlers** and select the three-dot menu for any crawler to access per-crawler actions.
34
+
4. Select multiple crawlers to use bulk copy buttons for user agents or detection IDs.
35
+
36
+
Learn more about [AI Crawl Control](/ai-crawl-control/).
Copy file name to clipboardExpand all lines: src/content/docs/ai-crawl-control/features/analyze-ai-traffic.mdx
+32-9Lines changed: 32 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -44,17 +44,28 @@ Identify traffic sources with referrer analytics to understand discovery pattern
44
44
- View top referrers driving traffic to your site.
45
45
- Understand discovery patterns and content popularity from AI operators.
46
46
47
-
### Track crawler requests over time
47
+
### Track successful crawler requests over time
48
48
49
-
Visualize crawler activity patterns over time using the **Requests over time** chart. You can group data by different dimensions to get more specific insights:
49
+
Visualize successful crawler activity patterns over time using the **Successful requests over time** chart, which shows requests with status codes 200-300. You can group data by different dimensions to get more specific insights:
|**Crawler**| Track activity from individual AI crawlers (like GPTBot, ClaudeBot, and Bytespider). |
54
+
|**Category**| Analyze crawlers by their purpose or type. |
55
+
|**Operator**| Discover which companies (such as OpenAI, Anthropic, and ByteDance) are crawling your site. |
56
+
|**Host**| Break down activity across multiple subdomains. |
57
+
58
+
### Monitor status code distribution
59
+
60
+
The **Status code distribution** chart shows HTTP response codes (2xx, 3xx, 4xx, 5xx) for AI crawler requests over time. This provides detailed visibility beyond the sparkline charts shown in the **Crawlers** tab.
61
+
62
+
The chart displays:
63
+
64
+
- Total request counts for each status code category (2xx, 3xx, 4xx, 5xx)
65
+
- Time series visualization showing status code trends over your selected time period
66
+
- Filtering by crawler, category, operator, and time range
67
+
68
+
Use this chart to identify patterns in how your site responds to AI crawlers in areas such as the distribution of error codes or the volume of redirects.
58
69
59
70
### Understand what content is crawled
60
71
@@ -94,3 +105,15 @@ Filter options:
94
105
</Tabs>
95
106
96
107
The values of the AI Crawl Control metrics will update according to your filter.
108
+
109
+
## Per-crawler drilldowns
110
+
111
+
The **Crawlers** tab includes an actions menu for each crawler with the following options:
|**View Metrics**| Filter the **Metrics** tab to the selected crawler. |
116
+
|**View on Cloudflare Radar**| Learn more about each verified crawler on Cloudflare Radar. |
117
+
|**View in Security Analytics**| Filter Security Analytics by detection IDs ([Bot Management](/bots/get-started/bot-management/) customers). |
118
+
|**Copy User Agent**| Copy user agent strings for use in [WAF custom rules](/waf/custom-rules/), [Redirect Rules](/rules/url-forwarding/), or robots.txt files. |
119
+
|**Copy Detection ID**| Copy detection IDs for use in [WAF custom rules](/waf/custom-rules/) ([Bot Management](/bots/get-started/bot-management/) customers). |
0 commit comments