- 
                Notifications
    
You must be signed in to change notification settings  - Fork 9.8k
 
Create changelog for updated AI Crawl Control metrics #25825
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 3 commits
c452863
              9b924ee
              2bb5eb2
              560f200
              8a51431
              File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,47 @@ | ||||||||||||||||||||||
| --- | ||||||||||||||||||||||
| title: Enhanced AI Crawl Control metrics with new drilldowns and filters | ||||||||||||||||||||||
| description: Analyze crawler activity with requests over time charts, referrer breakdowns, and configurable filters with data export capabilities | ||||||||||||||||||||||
| date: 2025-10-14 | ||||||||||||||||||||||
| --- | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| AI Crawl Control now provides enhanced metrics and CSV data exports to help you better understand AI crawler activity across your sites. | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| ## What's new | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| #### Track crawler requests over time | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| Visualize crawler activity patterns over time, and group data by different dimensions: | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| - **By Crawler** — Track activity from individual AI crawlers (GPTBot, ClaudeBot, Bytespider) | ||||||||||||||||||||||
| - **By Category** — Analyze crawler purpose or type | ||||||||||||||||||||||
| - **By Operator** — See which companies (OpenAI, Anthropic, ByteDance) are crawling your site | ||||||||||||||||||||||
                
      
                  CameronWhiteside marked this conversation as resolved.
               
              
                Outdated
          
            Show resolved
            Hide resolved
         | 
||||||||||||||||||||||
| - **By Host** — Break down activity across multiple subdomains | ||||||||||||||||||||||
| - **By Status Code** — Monitor HTTP response codes to crawlers (200s, 300s, 400s, 500s) | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
|  | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| #### Analyze referrer data (Paid plans) | ||||||||||||||||||||||
                
      
                  CameronWhiteside marked this conversation as resolved.
               
              
                Outdated
          
            Show resolved
            Hide resolved
         | 
||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| Identify traffic sources with referrer analytics: | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| - View top referrers driving traffic to your site | ||||||||||||||||||||||
| - Understand discovery patterns and content popularity from AI operators | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
|  | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| #### Export data | ||||||||||||||||||||||
                
      
                  CameronWhiteside marked this conversation as resolved.
               
              
                Outdated
          
            Show resolved
            Hide resolved
         | 
||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| Download your filtered view as a CSV: | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| - Includes all applied filters and groupings | ||||||||||||||||||||||
| - Useful for custom reporting and deeper analysis | ||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| ## Getting started | ||||||||||||||||||||||
                
      
                  CameronWhiteside marked this conversation as resolved.
               
              
                Outdated
          
            Show resolved
            Hide resolved
         | 
||||||||||||||||||||||
| 
     | 
||||||||||||||||||||||
| 1. Navigate to **AI Crawl Control** for any domain in your Cloudflare dashboard | ||||||||||||||||||||||
| 2. Select the **Metrics** tab | ||||||||||||||||||||||
| 3. Use the dimension tabs to explore different views of your data | ||||||||||||||||||||||
| 4. Apply filters to focus on specific crawlers, time ranges, or response codes | ||||||||||||||||||||||
| 5. Click **Download CSV** to export your filtered data for further analysis | ||||||||||||||||||||||
                
       | 
||||||||||||||||||||||
| 1. Navigate to **AI Crawl Control** for any domain in your Cloudflare dashboard | |
| 2. Select the **Metrics** tab | |
| 3. Use the dimension tabs to explore different views of your data | |
| 4. Apply filters to focus on specific crawlers, time ranges, or response codes | |
| 5. Click **Download CSV** to export your filtered data for further analysis | |
| 1. Log in to the Cloudflare dashboard, and select your account and domain. | |
| 2. Go to **AI Crawl Control** > **Metrics**. | |
| 3. Use the **Dimensions** tabs to explore different views of your data. | |
| 4. Apply filters to focus on specific crawlers, time ranges, or response codes. | |
| 5. Select **Download CSV** to export your filtered data for further analysis. | 
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated! (only think not kept was the Dimensions bold since the tabs aren't explicitly named, but I changed it to "grouping tabs" to make it more clear)
Uh oh!
There was an error while loading. Please reload this page.