|
| 1 | +--- |
| 2 | +title: Bot & Crawler Insights in Cloudflare Radar |
| 3 | +description: Expanded visibility into crawler, referral, and bot activity with new Radar API endpoints. |
| 4 | +products: |
| 5 | + - radar |
| 6 | +date: 2025-07-01T12:00:00Z |
| 7 | +--- |
| 8 | + |
| 9 | +### Web crawlers insights |
| 10 | + |
| 11 | +[**Radar**](/radar/) now offers expanded insights into web crawlers, giving you greater visibility into aggregated trends in crawl and refer activity. |
| 12 | + |
| 13 | +We have introduced the following endpoints: |
| 14 | + |
| 15 | +- [`/bots/crawlers/summary`](/api/resources/radar/subresources/bots/subresources/crawlers/subresources/summary/): Returns an overview of crawler HTTP request distributions across key dimensions. |
| 16 | +- [`/bots/crawlers/timeseries_group`](/api/resources/radar/subresources/bots/subresources/crawlers/subresources/timeseries_groups/): Provides time-series data on crawler request distributions across the same dimensions. |
| 17 | + |
| 18 | +These endpoints allow analysis across the following dimensions: |
| 19 | + |
| 20 | +- `user_agent`: Parsed data from the `User-Agent` header. |
| 21 | +- `referer`: Parsed data from the `Referer` header. |
| 22 | +- `crawl_refer_ratio`: Ratio of HTML page crawl requests to HTML page referrals by platform. |
| 23 | + |
| 24 | +### Broader bot insights |
| 25 | + |
| 26 | +In addition to crawler-specific insights, Radar now provides a broader set of bot endpoints: |
| 27 | + |
| 28 | +* [`/bots/`](/api/resources/radar/subresources/bots/): Lists all bots. |
| 29 | +* [`/bots/:bot_slug`](/api/resources/radar/subresources/bots/:bot_slug): Returns detailed metadata for a specific bot. |
| 30 | +* [`/bots/timeseries`](/api/resources/radar/subresources/bots/subresources/timeseries/): Time-series data for bot activity. |
| 31 | +* [`/bots/summary`](/api/resources/radar/subresources/bots/subresources/summary/): Returns an overview of bot HTTP request distributions across key dimensions. |
| 32 | +* [`/bots/timeseries_groups`](/api/resources/radar/subresources/bots/subresources/timeseries_groups/): Provides time-series data on bot request distributions across the same dimensions. |
| 33 | + |
| 34 | +These endpoints support filtering and breakdowns by: |
| 35 | + |
| 36 | +* `bot`: Bot name. |
| 37 | +* `bot_operator`: The organization or entity operating the bot. |
| 38 | +* `bot_category`: Classification of bot type. |
| 39 | + |
| 40 | +The previously available `verified_bots` endpoints have now been deprecated in favor of this set of bot insights APIs. |
| 41 | +While current data still focuses on verified bots, we plan to expand support for unverified bot traffic in the future. |
| 42 | + |
| 43 | +Learn more about the new Radar bot and crawler insights in our [blog post](https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar). |
0 commit comments