From 52be2429d2a79b378443eab04564c9e7ea7dd1c1 Mon Sep 17 00:00:00 2001 From: Andre Jesus Date: Fri, 30 May 2025 14:43:46 +0100 Subject: [PATCH 1/5] [Radar] Mention new crawlers endpoints in both release notes and changelog --- ...2025-06-03-radar-web-crawlers-insights.mdx | 23 +++++++++++++++++++ src/content/release-notes/radar.yaml | 4 ++++ 2 files changed, 27 insertions(+) create mode 100644 src/content/changelog/radar/2025-06-03-radar-web-crawlers-insights.mdx diff --git a/src/content/changelog/radar/2025-06-03-radar-web-crawlers-insights.mdx b/src/content/changelog/radar/2025-06-03-radar-web-crawlers-insights.mdx new file mode 100644 index 000000000000000..f6cd7a7ab3640bd --- /dev/null +++ b/src/content/changelog/radar/2025-06-03-radar-web-crawlers-insights.mdx @@ -0,0 +1,23 @@ +--- +title: Web Crawler Insights in Cloudflare Radar +description: Expanded visibility into crawler and referral activity with new Radar API endpoints. +products: + - radar +date: 2025-06-03T14:00:00Z +--- + +[**Radar**](/radar/) now offers expanded insights into web crawlers, giving you greater visibility into aggregated trends in crawl and refer activity. + +We have introduced the following endpoints: + +- [`summary`](/api/resources/radar/subresources/bots/subresources/crawlers/subresources/summary/): Returns an overview of crawler HTTP request distributions across key dimensions. +- [`timeseries_group`](/api/resources/radar/subresources/bots/subresources/crawlers/subresources/timeseries_groups/): Provides time-series data on crawler request distributions across the same dimensions. + +These endpoints allow analysis across the following dimensions: + +- `user_agent`: Parsed data from the `User-Agent` header. +- `referer`: Parsed data from the `Referer` header. +- `crawl_refer_ratio`: Ratio of HTML page crawl requests to HTML page referrals by platform. + + +Learn more about the new Radar crawler insights in our [blog post](https://blog.cloudflare.com/...). diff --git a/src/content/release-notes/radar.yaml b/src/content/release-notes/radar.yaml index ef9477c56bd3a97..2cc343a4a740130 100644 --- a/src/content/release-notes/radar.yaml +++ b/src/content/release-notes/radar.yaml @@ -5,6 +5,10 @@ productLink: "/radar/" productArea: Developer platform productAreaLink: /workers/platform/changelog/platform/ entries: + - publish_date: "2025-06-03" + title: Add web crawlers endpoints + description: |- + * Added [web crawlers endpoints](/api/resources/radar/subresources/bots/subresources/crawlers/). - publish_date: "2025-03-20" title: Endpoint deprecations and new BGP real-time routes endpoint description: |- From d4e1de84570f3d61295db6b6c9b28e796c43ebde Mon Sep 17 00:00:00 2001 From: Andre Jesus Date: Fri, 30 May 2025 14:28:16 +0100 Subject: [PATCH 2/5] [Radar] Add crawlers section to glossary --- src/content/docs/radar/glossary.mdx | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/content/docs/radar/glossary.mdx b/src/content/docs/radar/glossary.mdx index ff4f32c76421a50..71d6461aa3fe004 100644 --- a/src/content/docs/radar/glossary.mdx +++ b/src/content/docs/radar/glossary.mdx @@ -244,6 +244,14 @@ Each entry on the Verified Bots list exists because a corresponding IP address w The data displayed on domain-specific geographic traffic patterns is based solely on data from our recursive DNS services. All data displayed is in accordance with our privacy policies and commitments. This data may include attack traffic and cross-origin requests. +## Web crawlers + +[Web crawlers](https://www.cloudflare.com/learning/bots/what-is-a-web-crawler/) are a type of bot that browses the Internet to collect and index website content. They are used by search engines like Google or Bing to make pages discoverable in search results. + +They are also used by AI platforms, either to gather content for training large language models, or to retrieve up-to-date information for AI assistants. In both search and AI cases, crawlers work by following links from one page to another, creating a map of online content. + +Radar's crawl-to-refer ratio metric is calculated by first mapping crawl requests for HTML pages based on the `User-Agent` header, and referral requests for HTML pages based on the `Referer` header, by platform (e.g., the ratio for Google is based on crawl requests from Googlebot, and referral requests from Google platforms). As with other traffic metrics on Radar, the aggregation resolution for the ratio data is based on the length of the selected timeframe. Additionally, note that traffic referred by native apps may not include a `Referer` header. As such, because the referral counts only include traffic from Web-based tools, these calculations may overstate the respective ratios, but it is unclear by how much. + ## WHOIS WHOIS is a standard for publishing the contact and nameserver information for all registered domains. Each registrar maintains their own WHOIS service. Anyone can query the registrar's WHOIS service to reveal the data behind a given domain. From 11c7640a76c8a9dec524ff010f0480dec311e42b Mon Sep 17 00:00:00 2001 From: Andre Jesus Date: Fri, 27 Jun 2025 17:02:30 +0100 Subject: [PATCH 3/5] [Radar] Mention new bots endpoints in both release notes and changelog --- ...2025-06-03-radar-web-crawlers-insights.mdx | 23 ---------- .../radar/2025-07-01-radar-bots-insights.mdx | 43 +++++++++++++++++++ src/content/release-notes/radar.yaml | 8 +++- 3 files changed, 49 insertions(+), 25 deletions(-) delete mode 100644 src/content/changelog/radar/2025-06-03-radar-web-crawlers-insights.mdx create mode 100644 src/content/changelog/radar/2025-07-01-radar-bots-insights.mdx diff --git a/src/content/changelog/radar/2025-06-03-radar-web-crawlers-insights.mdx b/src/content/changelog/radar/2025-06-03-radar-web-crawlers-insights.mdx deleted file mode 100644 index f6cd7a7ab3640bd..000000000000000 --- a/src/content/changelog/radar/2025-06-03-radar-web-crawlers-insights.mdx +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: Web Crawler Insights in Cloudflare Radar -description: Expanded visibility into crawler and referral activity with new Radar API endpoints. -products: - - radar -date: 2025-06-03T14:00:00Z ---- - -[**Radar**](/radar/) now offers expanded insights into web crawlers, giving you greater visibility into aggregated trends in crawl and refer activity. - -We have introduced the following endpoints: - -- [`summary`](/api/resources/radar/subresources/bots/subresources/crawlers/subresources/summary/): Returns an overview of crawler HTTP request distributions across key dimensions. -- [`timeseries_group`](/api/resources/radar/subresources/bots/subresources/crawlers/subresources/timeseries_groups/): Provides time-series data on crawler request distributions across the same dimensions. - -These endpoints allow analysis across the following dimensions: - -- `user_agent`: Parsed data from the `User-Agent` header. -- `referer`: Parsed data from the `Referer` header. -- `crawl_refer_ratio`: Ratio of HTML page crawl requests to HTML page referrals by platform. - - -Learn more about the new Radar crawler insights in our [blog post](https://blog.cloudflare.com/...). diff --git a/src/content/changelog/radar/2025-07-01-radar-bots-insights.mdx b/src/content/changelog/radar/2025-07-01-radar-bots-insights.mdx new file mode 100644 index 000000000000000..3638e2c2c721eea --- /dev/null +++ b/src/content/changelog/radar/2025-07-01-radar-bots-insights.mdx @@ -0,0 +1,43 @@ +--- +title: Bot & Crawler Insights in Cloudflare Radar +description: Expanded visibility into crawler, referral, and bot activity with new Radar API endpoints. +products: + - radar +date: 2025-07-01T12:00:00Z +--- + +### Web crawlers insights + +[**Radar**](/radar/) now offers expanded insights into web crawlers, giving you greater visibility into aggregated trends in crawl and refer activity. + +We have introduced the following endpoints: + +- [`/bots/crawlers/summary`](/api/resources/radar/subresources/bots/subresources/crawlers/subresources/summary/): Returns an overview of crawler HTTP request distributions across key dimensions. +- [`/bots/crawlers/timeseries_group`](/api/resources/radar/subresources/bots/subresources/crawlers/subresources/timeseries_groups/): Provides time-series data on crawler request distributions across the same dimensions. + +These endpoints allow analysis across the following dimensions: + +- `user_agent`: Parsed data from the `User-Agent` header. +- `referer`: Parsed data from the `Referer` header. +- `crawl_refer_ratio`: Ratio of HTML page crawl requests to HTML page referrals by platform. + +### Broader bot insights + +In addition to crawler-specific insights, Radar now provides a broader set of bot endpoints: + +* [`/bots/`](/api/resources/radar/subresources/bots/): Lists all bots. +* [`/bots/:bot_slug`](/api/resources/radar/subresources/bots/:bot_slug): Returns detailed metadata for a specific bot. +* [`/bots/timeseries`](/api/resources/radar/subresources/bots/subresources/timeseries/): Time-series data for bot activity. +* [`/bots/summary`](/api/resources/radar/subresources/bots/subresources/summary/): Returns an overview of bot HTTP request distributions across key dimensions. +* [`/bots/timeseries_groups`](/api/resources/radar/subresources/bots/subresources/timeseries_groups/): Provides time-series data on bot request distributions across the same dimensions. + +These endpoints support filtering and breakdowns by: + +* `bot`: Bot name. +* `bot_operator`: The organization or entity operating the bot. +* `bot_category`: Classification of bot type. + +The previously available `verified_bots` endpoints have now been deprecated in favor of this set of bot insights APIs. +While current data still focuses on verified bots, we plan to expand support for unverified bot traffic in the future. + +Learn more about the new Radar bot and crawler insights in our [blog post](https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar). diff --git a/src/content/release-notes/radar.yaml b/src/content/release-notes/radar.yaml index 2cc343a4a740130..a5020065e73cd52 100644 --- a/src/content/release-notes/radar.yaml +++ b/src/content/release-notes/radar.yaml @@ -5,9 +5,13 @@ productLink: "/radar/" productArea: Developer platform productAreaLink: /workers/platform/changelog/platform/ entries: - - publish_date: "2025-06-03" - title: Add web crawlers endpoints + - publish_date: "2025-07-01" + title: Add Bots and Web Crawlers Endpoints description: |- + * Added new [bots endpoints](/api/resources/radar/subresources/bots/), replacing the deprecated verified bots + endpoints. Use the following replacements: + * `/verified_bots/top/bots` → `/bots/summary/bot` + * `/verified_bots/top/categories` → `/bots/summary/bot_category` * Added [web crawlers endpoints](/api/resources/radar/subresources/bots/subresources/crawlers/). - publish_date: "2025-03-20" title: Endpoint deprecations and new BGP real-time routes endpoint From b135c30b990ecfe28cf7f8cd170a58cb89010bb4 Mon Sep 17 00:00:00 2001 From: Andre Jesus Date: Fri, 27 Jun 2025 22:18:06 +0100 Subject: [PATCH 4/5] [Radar] Deprecate verified bots API --- .../release-notes/api-deprecations.yaml | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/src/content/release-notes/api-deprecations.yaml b/src/content/release-notes/api-deprecations.yaml index 3f09f3872155101..ac47045b6cc1333 100644 --- a/src/content/release-notes/api-deprecations.yaml +++ b/src/content/release-notes/api-deprecations.yaml @@ -5,6 +5,25 @@ productLink: "/fundamentals/" productArea: Core platform productAreaLink: /fundamentals/reference/changelog/platform/ entries: + - publish_date: "2025-07-01" + title: "Cloudflare Radar: Verified Bots APIs" + description: |- + Deprecation date: July 1, 2025 + + End of life date: January 1, 2026 + + The Radar Verified Bots API is now deprecated and will be replaced by the new Bots API. + + Deprecated APIs: + + * `GET /radar/verified_bots/top/bots` + * `GET /radar/verified_bots/top/categories` + + Replacements: + + * `GET /radar/bots/summary/bot` + * `GET /radar/bots/summary/category` + - publish_date: "2025-11-11" title: "Zero Trust Devices" description: |- From 0d2d2a0d3967fc210815ddc6ef1f9dfc62666aff Mon Sep 17 00:00:00 2001 From: Andre Jesus Date: Mon, 30 Jun 2025 22:59:14 +0100 Subject: [PATCH 5/5] Fix links --- .../radar/2025-07-01-radar-bots-insights.mdx | 12 ++++++------ src/content/release-notes/radar.yaml | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/content/changelog/radar/2025-07-01-radar-bots-insights.mdx b/src/content/changelog/radar/2025-07-01-radar-bots-insights.mdx index 3638e2c2c721eea..885d274c2fcc832 100644 --- a/src/content/changelog/radar/2025-07-01-radar-bots-insights.mdx +++ b/src/content/changelog/radar/2025-07-01-radar-bots-insights.mdx @@ -12,8 +12,8 @@ date: 2025-07-01T12:00:00Z We have introduced the following endpoints: -- [`/bots/crawlers/summary`](/api/resources/radar/subresources/bots/subresources/crawlers/subresources/summary/): Returns an overview of crawler HTTP request distributions across key dimensions. -- [`/bots/crawlers/timeseries_group`](/api/resources/radar/subresources/bots/subresources/crawlers/subresources/timeseries_groups/): Provides time-series data on crawler request distributions across the same dimensions. +- [`/bots/crawlers/summary`](/api/resources/radar/subresources/bots/subresources/web_crawlers/methods/summary/): Returns an overview of crawler HTTP request distributions across key dimensions. +- [`/bots/crawlers/timeseries_group`](/api/resources/radar/subresources/bots/subresources/web_crawlers/methods/timeseries_groups/): Provides time-series data on crawler request distributions across the same dimensions. These endpoints allow analysis across the following dimensions: @@ -26,10 +26,10 @@ These endpoints allow analysis across the following dimensions: In addition to crawler-specific insights, Radar now provides a broader set of bot endpoints: * [`/bots/`](/api/resources/radar/subresources/bots/): Lists all bots. -* [`/bots/:bot_slug`](/api/resources/radar/subresources/bots/:bot_slug): Returns detailed metadata for a specific bot. -* [`/bots/timeseries`](/api/resources/radar/subresources/bots/subresources/timeseries/): Time-series data for bot activity. -* [`/bots/summary`](/api/resources/radar/subresources/bots/subresources/summary/): Returns an overview of bot HTTP request distributions across key dimensions. -* [`/bots/timeseries_groups`](/api/resources/radar/subresources/bots/subresources/timeseries_groups/): Provides time-series data on bot request distributions across the same dimensions. +* [`/bots/:bot_slug`](/api/resources/radar/subresources/bots/methods/get/): Returns detailed metadata for a specific bot. +* [`/bots/timeseries`](/api/resources/radar/subresources/bots/methods/timeseries/): Time-series data for bot activity. +* [`/bots/summary`](/api/resources/radar/subresources/bots/methods/summary/): Returns an overview of bot HTTP request distributions across key dimensions. +* [`/bots/timeseries_groups`](/api/resources/radar/subresources/bots/methods/timeseries_groups/): Provides time-series data on bot request distributions across the same dimensions. These endpoints support filtering and breakdowns by: diff --git a/src/content/release-notes/radar.yaml b/src/content/release-notes/radar.yaml index a5020065e73cd52..9db9834b2fc04fb 100644 --- a/src/content/release-notes/radar.yaml +++ b/src/content/release-notes/radar.yaml @@ -12,7 +12,7 @@ entries: endpoints. Use the following replacements: * `/verified_bots/top/bots` → `/bots/summary/bot` * `/verified_bots/top/categories` → `/bots/summary/bot_category` - * Added [web crawlers endpoints](/api/resources/radar/subresources/bots/subresources/crawlers/). + * Added [web crawlers endpoints](/api/resources/radar/subresources/bots/subresources/web_crawlers/). - publish_date: "2025-03-20" title: Endpoint deprecations and new BGP real-time routes endpoint description: |-