[Radar] Add internet services ranking, robots.txt and workers AI entries to glossary

andre-j3sus · andre-j3sus · commit 5fc2fb2365ac · 2025-01-22T16:50:18.000Z
* Fix occurrences of 'Netflows' to 'NetFlows'
* Fix json blocks with API response examples
diff --git a/.gitignore b/.gitignore
@@ -25,4 +25,6 @@ pnpm-debug.log*
 
 .wrangler
 /assets/secrets
-/worker/functions/
+/worker/functions/
+
+.idea
diff --git a/src/content/docs/radar/get-started/making-comparisons.mdx b/src/content/docs/radar/get-started/making-comparisons.mdx
@@ -8,7 +8,7 @@ sidebar:
 
 When comparing time series, across locations/time ranges/etc., in endpoints that normalize values using [min-max](/radar/concepts/normalization), you must do so in the same request. This is done by asking for multiple series. All values will then be normalized using the same minimum and maximum value and can safely be compared against each other.
 
-[Netflows](/radar/investigate/netflows) values are normalized using [min0-max](/radar/concepts/normalization), so we will use it as an example. Refer to [Get NetFlow time series](/api/resources/radar/subresources/netflows/methods/timeseries/) for more information.
+[NetFlows](/radar/investigate/netflows) values are normalized using [min0-max](/radar/concepts/normalization), so we will use it as an example. Refer to [Get NetFlow time series](/api/resources/radar/subresources/netflows/methods/timeseries/) for more information.
 
 ## Compare locations
 
@@ -49,7 +49,7 @@ The response (shortened below for brevity) uses the provided `name` property to
         "startTime": "2022-10-26T17:00:00Z",
         "endTime": "2022-11-02T17:00:00Z"
       },
-      "aggInterval": "ONE_HOUR",
+      "aggInterval": "ONE_HOUR"
     }
   }
 }
@@ -113,13 +113,19 @@ In the above example, we requested for an [aggregation interval](/radar/concepts
 The `result` property should return a response like this:
 
 ```json
-"tonga": {
-  "timestamps": ["2022-10-15T02:00:00Z", "2022-10-15T03:00:00Z", "2022-10-15T04:00:00Z", "2022-10-15T05:00:00Z"],
-  "values": ["1.0", "0.832473", "0.820083", "0.79408"]
-},
-"tonga_outage": {
-  "timestamps": ["2022-01-15T02:00:00Z", "2022-01-15T03:00:00Z", "2022-01-15T04:00:00Z", "2022-01-15T05:00:00Z"],
-  "values": ["0.354105", "0.357287", "0.181811", "0.044198"]
+{
+  "success": true,
+  "errors": [],
+  "result": {
+		"tonga": {
+			"timestamps": ["2022-10-15T02:00:00Z", "2022-10-15T03:00:00Z", "2022-10-15T04:00:00Z", "2022-10-15T05:00:00Z"],
+			"values": ["1.0", "0.832473", "0.820083", "0.79408"]
+		},
+		"tonga_outage": {
+			"timestamps": ["2022-01-15T02:00:00Z", "2022-01-15T03:00:00Z", "2022-01-15T04:00:00Z", "2022-01-15T05:00:00Z"],
+			"values": ["0.354105", "0.357287", "0.181811", "0.044198"]
+		}
+  }
 }
 ```
 
diff --git a/src/content/docs/radar/glossary.mdx b/src/content/docs/radar/glossary.mdx
@@ -110,6 +110,10 @@ Internet connectivity can experience outages or disruptions due to a number of f
 
 An early warning signal that an Internet outage may be underway on a given network or in a given country is an anomalous drop in traffic as compared to historical traffic patterns and trends. Internet anomalies listed in the Cloudflare Radar Outage Center represent an algorithmically-observed anomalous drop in traffic for the listed entity. If a given entry is marked as verified, it means that we have manually corroborated the observed drop in traffic across multiple Cloudflare data sources and/or third-party sources such as [IODA](https://ioda.inetintel.cc.gatech.edu/), or third-party sources of information, such as those listed above. In the case of the latter, an associated Internet outage event will be opened, with the event listed in the Internet Outages table (and API).
 
+## Internet services ranking
+
+Internet services ranking is based on our anonymized and aggregated 1.1.1.1 resolver data, complies with our [privacy policy](https://www.cloudflare.com/en-gb/privacypolicy/), and aims to identify the top most popular Internet services that reflect how people use the Internet globally. A service represents one or more domains aggregated together. Ranking popularity metric is best described as the estimated number of unique users that access domains associated with a service, over some period of time.
+
 ## Internet traffic trends
 
 Trends observed in Internet traffic originating globally or within a given location or autonomous system within the selected time range, based on aggregated data from our network.
@@ -154,6 +158,14 @@ Network-level DDoS attacks graphs are based on traffic measured in bytes.
 
 The Post-Quantum Encryption Adoption graph shows the share of HTTPS requests to Cloudflare that are encrypted with post-quantum (PQ) cryptography. Additional details about Cloudflare's support for PQ cryptography can be found at [Cloudflare Research](https://pq.cloudflareresearch.com/).
 
+## Robots.txt
+
+A [robots.txt](https://www.cloudflare.com/learning/bots/what-is-robots-txt/) file contains instructions for bots that tell them which webpages they can and cannot access.
+
+The data displayed for robots.txt is based on successfully parsed robots.txt files from the top 10,000 domains (should we link to somewhere). From these files, we count the occurrences of each user agent under the "Allow" and "Disallow" directives. A user agent is classified as "fully allowed" or "fully disallowed" if the directive value is "*". Otherwise, if the user agent is only allowed or disallowed to crawl specific paths, it is classified as "partially allowed" or "partially disallowed."
+
+Currently, we only include AI-focused user agents listed in the [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) repository.
+
 ## TCP resets and timeouts
 
 In the Transmission Control Protocol (TCP), client-initiated connection resets (via the RST flag, TCP's "panic button") are atypical, and indicate to the server that *something went wrong* requiring the connection to be closed immediately. Similarly, connection timeouts (where the server closes a connection due to an unresponsive client) should not happen in conventional data exchanges. For comparison, a typical TCP connection consists of a 3-way handshake initiated by a client with a SYN packet to the server, then a data exchange moderated with ACK and PSH flags in the data packets, and finally a graceful close initiated from either side with a FIN packet. A FIN close is considered graceful because it ensures both sides complete their data transfer before closing the connection. In contrast, a timeout or RST flag triggers a hard stop, even if data is waiting to be sent or acknowledged. See [RFC 9293](https://datatracker.ietf.org/doc/html/rfc9293) for more details on the TCP protocol.
@@ -205,3 +217,7 @@ The data displayed on domain-specific geographic traffic patterns is based solel
 ## WHOIS
 
 WHOIS is a standard for publishing the contact and nameserver information for all registered domains. Each registrar maintains their own WHOIS service. Anyone can query the registrar's WHOIS service to reveal the data behind a given domain.
+
+## Workers AI
+
+[Workers AI](https://developers.cloudflare.com/workers-ai) allows you to run machine learning models, on the Cloudflare network, from your own code -- whether that be from Workers, Pages, or anywhere via the Cloudflare API. The data displayed for Workers AI is based on the number of Cloudflare accounts using a model during a specific time interval.
diff --git a/src/content/docs/radar/investigate/http-requests.mdx b/src/content/docs/radar/investigate/http-requests.mdx
@@ -6,7 +6,7 @@ sidebar:
 
 ---
 
-While in [Netflows](/radar/investigate/netflows/) we can inspect bytes and packets reaching Cloudflare's edge routers, in HTTP requests we are a layer above in the [OSI model](https://en.wikipedia.org/wiki/OSI_model). HTTP requests examines complete HTTP requests from end users that reach websites served by Cloudflare's [CDN](https://www.cloudflare.com/en-gb/learning/cdn/what-is-a-cdn/).
+While in [NetFlows](/radar/investigate/netflows/) we can inspect bytes and packets reaching Cloudflare's edge routers, in HTTP requests we are a layer above in the [OSI model](https://en.wikipedia.org/wiki/OSI_model). HTTP requests examines complete HTTP requests from end users that reach websites served by Cloudflare's [CDN](https://www.cloudflare.com/en-gb/learning/cdn/what-is-a-cdn/).
 
 :::note
 
@@ -87,15 +87,21 @@ curl "https://api.cloudflare.com/client/v4/radar/http/summary/device_type?name=h
 Here is the abbreviated response:
 
 ```json
-"human": {
-  "mobile": "54.967243",
-  "desktop": "44.974006",
-  "other": "0.058751"
-},
-"bot": {
-  "desktop": "83.275452",
-  "mobile": "16.707455",
-  "other": "0.017093"
+{
+  "success": true,
+  "errors": [],
+  "result": {
+		"human": {
+			"mobile": "54.967243",
+			"desktop": "44.974006",
+			"other": "0.058751"
+		},
+		"bot": {
+			"desktop": "83.275452",
+			"mobile": "16.707455",
+			"other": "0.017093"
+		}
+  }
 }
 ```
 
@@ -113,13 +119,19 @@ curl "https://api.cloudflare.com/client/v4/radar/http/summary/ip_version?name=hu
 This returns the following:
 
 ```json
-"human": {
-  "IPv4": "76.213647",
-  "IPv6": "23.786353"
-},
-"bot": {
-  "IPv4": "91.492032",
-  "IPv6": "8.507968"
+{
+  "success": true,
+  "errors": [],
+  "result": {
+		"human": {
+			"IPv4": "76.213647",
+			"IPv6": "23.786353"
+		},
+		"bot": {
+			"IPv4": "91.492032",
+			"IPv6": "8.507968"
+		}
+  }
 }
 ```
 
@@ -141,29 +153,35 @@ curl "https://api.cloudflare.com/client/v4/radar/http/top/locations/ip_version/I
 ```
 
 ```json
-"ipv6": [
-  {
-    "clientCountryAlpha2": "IN",
-    "clientCountryName": "India",
-    "value": "50.612747"
-  }, {
-    "clientCountryAlpha2": "MY",
-    "clientCountryName": "Malaysia",
-    "value": "46.233654"
-  }, {
-    "clientCountryAlpha2": "UY",
-    "clientCountryName": "Uruguay",
-    "value": "39.796762"
-  }, {
-    "clientCountryAlpha2": "LK",
-    "clientCountryName": "Sri Lanka",
-    "value": "39.709355"
-  }, {
-    "clientCountryAlpha2": "VN",
-    "clientCountryName": "Vietnam",
-    "value": "39.1514"
+{
+  "success": true,
+  "errors": [],
+  "result": {
+		"ipv6": [
+			{
+				"clientCountryAlpha2": "IN",
+				"clientCountryName": "India",
+				"value": "50.612747"
+			}, {
+				"clientCountryAlpha2": "MY",
+				"clientCountryName": "Malaysia",
+				"value": "46.233654"
+			}, {
+				"clientCountryAlpha2": "UY",
+				"clientCountryName": "Uruguay",
+				"value": "39.796762"
+			}, {
+				"clientCountryAlpha2": "LK",
+				"clientCountryName": "Sri Lanka",
+				"value": "39.709355"
+			}, {
+				"clientCountryAlpha2": "VN",
+				"clientCountryName": "Vietnam",
+				"value": "39.1514"
+			}
+		]
   }
-]
+}
 ```
 
 According to the returned data, India is leading in IPv6 adoption.
diff --git a/src/content/docs/radar/investigate/index.mdx b/src/content/docs/radar/investigate/index.mdx
@@ -8,6 +8,6 @@ sidebar:
 
 import { DirectoryListing } from "~/components"
 
-Explore the different types of data available in Cloudflare Radar, from Netflows (what feeds the [Internet Traffic Change](https://radar.cloudflare.com/) chart in Radar) to HTTP requests and DNS queries.
+Explore the different types of data available in Cloudflare Radar, from NetFlows (what feeds the [Internet Traffic Change](https://radar.cloudflare.com/) chart in Radar) to HTTP requests and DNS queries.
 
 <DirectoryListing />
diff --git a/src/content/docs/radar/investigate/netflows.mdx b/src/content/docs/radar/investigate/netflows.mdx
@@ -1,14 +1,14 @@
 ---
 pcx_content_type: reference
-title: Netflows
+title: NetFlows
 sidebar:
   order: 2
 
 ---
 
-[Netflows](https://en.wikipedia.org/wiki/NetFlow) shows network traffic data from end users collected from Cloudflare's edge routers. Netflows' data also feeds the [Internet traffic change](https://radar.cloudflare.com/) chart.
+[NetFlows](https://en.wikipedia.org/wiki/NetFlow) shows network traffic data from end users collected from Cloudflare's edge routers. NetFlows' data also feeds the [Internet traffic change](https://radar.cloudflare.com/) chart.
 
-Netflows includes all types of traffic from Cloudflare's routers, not just traffic to websites served by Cloudflare's [CDN](https://www.cloudflare.com/en-gb/learning/cdn/what-is-a-cdn/).
+NetFlows includes all types of traffic from Cloudflare's routers, not just traffic to websites served by Cloudflare's [CDN](https://www.cloudflare.com/en-gb/learning/cdn/what-is-a-cdn/).
 
 ## List of endpoints
 
@@ -20,7 +20,7 @@ Besides comparing time series across locations or date ranges (discussed in [Mak
 
 :::note[NetFlow products]
 
-`HTTP` traffic only includes web traffic to Cloudflare's zones, while `ALL` also includes traffic to all other services, like [Spectrum](/spectrum/), [Magic Transit](/magic-transit/), [1.1.1.1](/1.1.1.1/), and others. 
+`HTTP` traffic only includes web traffic to Cloudflare's zones, while `ALL` also includes traffic to all other services, like [Spectrum](/spectrum/), [Magic Transit](/magic-transit/), [1.1.1.1](/1.1.1.1/), and others.
 :::
 
 In the following example, we will examine both `ALL` and `HTTP` traffic in two [autonomous systems](https://www.cloudflare.com/en-gb/learning/network-layer/what-is-an-autonomous-system/). First, we will examine [AS3243](https://radar.cloudflare.com/as3243), a Portuguese local Internet Service Provider (ISP). The parameters for all traffic are `name=AS3243_all&product=ALL&dateRange=1d&asn=3243`, and for just the HTTP traffic are `name=AS3243_http&product=HTTP&dateRange=1d&asn=3243`):
@@ -33,14 +33,20 @@ curl "https://api.cloudflare.com/client/v4/radar/netflows/timeseries?name=meo_al
 This is the abbreviated response:
 
 ```json
-"AS3243_all": {
-  "timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
-  "values": ["0.565885", "0.586434", "..."]
-},
-"AS3243_http": {
-  "timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
-  "values": ["0.548564", "0.568329", "..."]
-},
+{
+  "success": true,
+  "errors": [],
+  "result": {
+		"AS3243_all": {
+			"timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
+			"values": ["0.565885", "0.586434", "..."]
+		},
+		"AS3243_http": {
+			"timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
+			"values": ["0.548564", "0.568329", "..."]
+		}
+	}
+}
 ```
 
 `HTTP` traffic values are similar to `ALL` traffic values. This means that most traffic Cloudflare receives from this AS is traffic to websites served by Cloudflare's [CDN](https://www.cloudflare.com/en-gb/learning/cdn/what-is-a-cdn/) product.
@@ -55,13 +61,19 @@ curl "https://api.cloudflare.com/client/v4/radar/netflows/timeseries?name=AS174_
 The abbreviated response is:
 
 ```json
-"AS174_all": {
-  "timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
-  "values": ["0.917348", "1.0", "..."]
-},
-"AS174_http": {
-  "timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
-  "values": ["0.381777", "0.408091", "..."]
+{
+  "success": true,
+  "errors": [],
+  "result": {
+		"AS174_all": {
+			"timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
+			"values": ["0.917348", "1.0", "..."]
+		},
+		"AS174_http": {
+			"timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
+			"values": ["0.381777", "0.408091", "..."]
+		}
+	}
 }
 ```
 
@@ -77,21 +89,27 @@ curl "https://api.cloudflare.com/client/v4/radar/netflows/timeseries?name=AS174_
 which would lead to a response like this:
 
 ```json
-"AS174_all": {
-  "timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
-  "values": ["0.917348", "1.0", "..."]
-},
-"AS174_http": {
-  "timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
-  "values": ["0.381777", "0.408091", "..."]
-},
-"AS3243_all": {
-  "timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
-  "values": ["0.317136", "0.328652", "..."]
-},
-"AS3243_http": {
-  "timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
-  "values": ["0.307429", "0.318505", "..."]
+{
+  "success": true,
+  "errors": [],
+  "result": {
+		"AS174_all": {
+			"timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
+			"values": ["0.917348", "1.0", "..."]
+		},
+		"AS174_http": {
+			"timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
+			"values": ["0.381777", "0.408091", "..."]
+		},
+		"AS3243_all": {
+			"timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
+			"values": ["0.317136", "0.328652", "..."]
+		},
+		"AS3243_http": {
+			"timestamps": ["2022-11-08T14:00:00Z", "2022-11-08T15:00:00Z", "..."],
+			"values": ["0.307429", "0.318505", "..."]
+		}
+	}
 }
 ```
 
diff --git a/src/content/docs/radar/investigate/outages.mdx b/src/content/docs/radar/investigate/outages.mdx
