|
| 1 | +--- |
| 2 | +title: Enhanced AI Crawl Control metrics with new drilldowns and filters |
| 3 | +description: Analyze crawler activity with requests over time charts, referrer breakdowns, and configurable filters with data export capabilities |
| 4 | +date: 2025-10-14 |
| 5 | +--- |
| 6 | + |
| 7 | +AI Crawl Control now provides enhanced metrics and CSV data exports to help you better understand AI crawler activity across your sites. |
| 8 | + |
| 9 | +## What's new |
| 10 | + |
| 11 | +### Track crawler requests over time |
| 12 | + |
| 13 | +Visualize crawler activity patterns over time, and group data by different dimensions: |
| 14 | + |
| 15 | +- **By Crawler** — Track activity from individual AI crawlers (GPTBot, ClaudeBot, Bytespider) |
| 16 | +- **By Category** — Analyze crawler purpose or type |
| 17 | +- **By Operator** — Discover which companies (OpenAI, Anthropic, ByteDance) are crawling your site |
| 18 | +- **By Host** — Break down activity across multiple subdomains |
| 19 | +- **By Status Code** — Monitor HTTP response codes to crawlers (200s, 300s, 400s, 500s) |
| 20 | + |
| 21 | + |
| 22 | + |
| 23 | +### Analyze referrer data (Paid plans) |
| 24 | + |
| 25 | +Identify traffic sources with referrer analytics: |
| 26 | + |
| 27 | +- View top referrers driving traffic to your site |
| 28 | +- Understand discovery patterns and content popularity from AI operators |
| 29 | + |
| 30 | + |
| 31 | + |
| 32 | +### Export data |
| 33 | + |
| 34 | +Download your filtered view as a CSV: |
| 35 | + |
| 36 | +- Includes all applied filters and groupings |
| 37 | +- Useful for custom reporting and deeper analysis |
| 38 | + |
| 39 | +## Get started |
| 40 | + |
| 41 | +1. Log in to the Cloudflare dashboard, and select your account and domain. |
| 42 | +2. Go to **AI Crawl Control** > **Metrics**. |
| 43 | +3. Use the grouping tabs to explore different views of your data. |
| 44 | +4. Apply filters to focus on specific crawlers, time ranges, or response codes. |
| 45 | +5. Select **Download CSV** to export your filtered data for further analysis. |
| 46 | + |
| 47 | +Learn more about [AI Crawl Control](/ai-crawl-control). |
0 commit comments