cloudflare · andre-j3sus · Jul 1, 2025 · May 30, 2025 · May 30, 2025 · Jun 27, 2025
@@ -0,0 +1,43 @@
+---
+title: Bot & Crawler Insights in Cloudflare Radar
+description: Expanded visibility into crawler, referral, and bot activity with new Radar API endpoints.
+products:
+  - radar
+date: 2025-07-01T12:00:00Z
+---
+
+### Web crawlers insights
+
+[**Radar**](/radar/) now offers expanded insights into web crawlers, giving you greater visibility into aggregated trends in crawl and refer activity.
+
+We have introduced the following endpoints:
+
+- [`/bots/crawlers/summary`](/api/resources/radar/subresources/bots/subresources/web_crawlers/methods/summary/): Returns an overview of crawler HTTP request distributions across key dimensions.
+- [`/bots/crawlers/timeseries_group`](/api/resources/radar/subresources/bots/subresources/web_crawlers/methods/timeseries_groups/): Provides time-series data on crawler request distributions across the same dimensions.
+
+These endpoints allow analysis across the following dimensions:
+
+- `user_agent`: Parsed data from the `User-Agent` header.
+- `referer`: Parsed data from the `Referer` header.
+- `crawl_refer_ratio`: Ratio of HTML page crawl requests to HTML page referrals by platform.
+
+### Broader bot insights
+
+In addition to crawler-specific insights, Radar now provides a broader set of bot endpoints:
+
+* [`/bots/`](/api/resources/radar/subresources/bots/): Lists all bots.
+* [`/bots/:bot_slug`](/api/resources/radar/subresources/bots/methods/get/): Returns detailed metadata for a specific bot.
+* [`/bots/timeseries`](/api/resources/radar/subresources/bots/methods/timeseries/): Time-series data for bot activity.
+* [`/bots/summary`](/api/resources/radar/subresources/bots/methods/summary/): Returns an overview of bot HTTP request distributions across key dimensions.
+* [`/bots/timeseries_groups`](/api/resources/radar/subresources/bots/methods/timeseries_groups/): Provides time-series data on bot request distributions across the same dimensions.
+
+These endpoints support filtering and breakdowns by:
+
+* `bot`: Bot name.
+* `bot_operator`: The organization or entity operating the bot.
+* `bot_category`: Classification of bot type.
+
+The previously available `verified_bots` endpoints have now been deprecated in favor of this set of bot insights APIs.
+While current data still focuses on verified bots, we plan to expand support for unverified bot traffic in the future.
+
+Learn more about the new Radar bot and crawler insights in our [blog post](https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar).
@@ -244,6 +244,14 @@ Each entry on the Verified Bots list exists because a corresponding IP address w
 
 The data displayed on domain-specific geographic traffic patterns is based solely on data from our recursive DNS services. All data displayed is in accordance with our privacy policies and commitments. This data may include attack traffic and cross-origin requests.
 
+## Web crawlers
+
+[Web crawlers](https://www.cloudflare.com/learning/bots/what-is-a-web-crawler/) are a type of bot that browses the Internet to collect and index website content. They are used by search engines like Google or Bing to make pages discoverable in search results.
+
+They are also used by AI platforms, either to gather content for training large language models, or to retrieve up-to-date information for AI assistants. In both search and AI cases, crawlers work by following links from one page to another, creating a map of online content.
+
+Radar's crawl-to-refer ratio metric is calculated by first mapping crawl requests for HTML pages based on the `User-Agent` header, and referral requests for HTML pages based on the `Referer` header, by platform (e.g., the ratio for Google is based on crawl requests from Googlebot, and referral requests from Google platforms). As with other traffic metrics on Radar, the aggregation resolution for the ratio data is based on the length of the selected timeframe. Additionally, note that traffic referred by native apps may not include a `Referer` header. As such, because the referral counts only include traffic from Web-based tools, these calculations may overstate the respective ratios, but it is unclear by how much.
+
 ## WHOIS
 
 WHOIS is a standard for publishing the contact and nameserver information for all registered domains. Each registrar maintains their own WHOIS service. Anyone can query the registrar's WHOIS service to reveal the data behind a given domain.

@@ -5,6 +5,25 @@ productLink: "/fundamentals/"
 productArea: Core platform
 productAreaLink: /fundamentals/reference/changelog/platform/
 entries:
+  - publish_date: "2025-07-01"
+    title: "Cloudflare Radar: Verified Bots APIs"
+    description: |-
+      Deprecation date: July 1, 2025
+
+      End of life date: January 1, 2026
+
+      The Radar Verified Bots API is now deprecated and will be replaced by the new Bots API.
+
+      Deprecated APIs:
+
+        * `GET /radar/verified_bots/top/bots`
+        * `GET /radar/verified_bots/top/categories`
+
+      Replacements:
+
+        * `GET /radar/bots/summary/bot`
+        * `GET /radar/bots/summary/category`
+
   - publish_date: "2025-11-11"
     title: "Zero Trust Devices"
     description: |-

@@ -5,6 +5,14 @@ productLink: "/radar/"
 productArea: Developer platform
 productAreaLink: /workers/platform/changelog/platform/
 entries:
+  - publish_date: "2025-07-01"
+    title: Add Bots and Web Crawlers Endpoints
+    description: |-
+      * Added new [bots endpoints](/api/resources/radar/subresources/bots/), replacing the deprecated verified bots
+      endpoints. Use the following replacements:
+        * `/verified_bots/top/bots` → `/bots/summary/bot`
+        * `/verified_bots/top/categories` → `/bots/summary/bot_category`
+      * Added [web crawlers endpoints](/api/resources/radar/subresources/bots/subresources/web_crawlers/).
   - publish_date: "2025-03-20"
     title: Endpoint deprecations and new BGP real-time routes endpoint
     description: |-