[Bots] Web Bot Auth docs

public/__redirects

@@ -238,6 +238,7 @@
 /bots/get-started/bm-subscription/ /bots/get-started/bot-management/ 301
 /bots/get-started/pro/ /bots/get-started/super-bot-fight-mode/ 301
 /bots/additional-configurations/javascript-detections/ /cloudflare-challenges/challenge-types/javascript-detections/ 301
+/bots/troubleshooting/frequently-asked-questions/ /bots/frequently-asked-questions/ 301
 
 #browser-rendering
 /browser-rendering/get-started/browser-rendering-with-do/ /browser-rendering/workers-binding-api/browser-rendering-with-do/ 301

src/content/docs/bots/concepts/bot/index.mdx

@@ -23,6 +23,8 @@ For more background, refer to [What is a bot?](https://www.cloudflare.com/learni
 
 <Render file="verified-bots" />
 
+For more information, refer to [Verified bots](/bots/concepts/bot/verified-bots/overview/).
+
 :::note
 
 The method for allowing or blocking verified bots depends on [your plan](/bots/get-started/).

src/content/docs/bots/concepts/bot/verified-bots/categories.mdx

@@ -2,7 +2,7 @@
 pcx_content_type: reference
 title: Verified bot categories
 sidebar:
-    order: 3
+    order: 20
     label: Categories
 
 ---
@@ -11,7 +11,7 @@ You can segment your verified bot traffic by its type and purpose by adding the
 
 :::note
 
-The Verified Bot Categories field is not compatible with legacy Firewall rules. 
+The Verified Bot Categories field is not compatible with legacy Firewall rules.
 :::
 
 ## Categories

src/content/docs/bots/concepts/bot/verified-bots/ip-validation.mdx

@@ -0,0 +1,57 @@
+---
+pcx_content_type: concept
+title: IP validation
+sidebar:
+    order: 7
+    label: IP validation
+
+---
+
+import { GlossaryTooltip, Steps } from "~/components"
+
+The IP validation method aims to identify all of the IP addresses that a bot may use to send requests.
+
+Cloudflare can achieve this in two ways:
+
+- **Using IP list provided by the bot owner**: The bot owner can host a public list of IP ranges (for example, [Googlebot's list](https://developers.google.com/static/search/apis/ipranges/googlebot.json)). Cloudflare fetches and uses this list directly for validation.
+- **Using Domain-based reverse DNS**: The bot owner can provide a domain (or set of domains) that their bot requests originate from. Cloudflare collects the IP addresses observed in the requests with the bot's user agent, and performs reverse DNS lookups. If the reverse DNS of an IP resolves to one of the provided domains, Cloudflare considers it valid and stores it.
+
+## Public IP List
+
+To verify a bot using a public IP list, you need to provide:
+
+- A fixed and limited set of IP addresses, which can be verified via publicly accessible plain-text, `JSON`, or `CSV`.
+- IP addresses used solely by the bot owner.
+- A user-agent match pattern.
+
+## Reverse DNS
+
+To verify a bot using reverse DNS, you need to provide:
+
+- A list of domain suffixes to validate DNS records.
+- IP addresses should have PTR records set correctly.
+- A user-agent match pattern.
+
+## Generic user-agents
+
+User-agent patterns that match generic user-agents will be rejected by the Verified Bots API. When you add a user-agent pattern that is considered very common to the Verified Bot form, you may encounter an error message that will prompt you to correct the user-agent before you can submit again.
+
+Generic user-agents include:
+
+- `Dart`
+- `Go-http-client`
+- `GuzzleHttp`
+- `Google Chrome`
+- `Mozilla Firefox`
+- `Safari`
+- `Nessus`
+- `Websocket++`
+- `cloudflare-go`
+- `fasthttp`
+- `got`
+- `nginx-ssl early hints`
+- `node`
+- `node-fetch`
+- `okhttp`
+- `python-requests`
+- `uTorrent`

src/content/docs/bots/concepts/bot/verified-bots/overview.mdx

@@ -0,0 +1,37 @@
+---
+pcx_content_type: concept
+title: Overview
+sidebar:
+    order: 3
+    label: Overview
+
+---
+
+import { GlossaryTooltip } from "~/components"
+
+A **verified bot** is a bot which has been added to Cloudflare's list of <GlossaryTooltip term="verified bot">verified bots</GlossaryTooltip>.
+
+You can request for your bot to be added to Cloudflare's list of verified bots by filling out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard.
+
+## Verified bot requirement
+
+For a bot to be verified, it must meet the following requirements:
+
+1. The bot must follow [verified bots policy](/bots/concepts/bot/verified-bots/policy/).
+2. The bot must be verified using one of the following verification methods:
+   - [Web Bot Auth](/bots/concepts/bot/verified-bots/web-bot-auth/)
+   - [IP validation](/bots/concepts/bot/verified-bots/ip-validation/)
+
+Once Cloudflare verifies a bot, it will appear on the [Cloudflare Radar's list of verified bots](https://radar.cloudflare.com/verified-bots).
+
+:::note
+Bot operators who prefer not to create a free Cloudflare account can do so using our [old form](https://docs.google.com/forms/d/e/1FAIpQLSdqYNuULEypMnp4i5pROSc-uP6x65Xub9svD27mb8JChA_-XA/viewform?usp=sf_link), but the waiting time is up to several weeks for verified bot requests to be evaluated.
+:::
+
+## Transient false negatives
+
+Once Cloudflare lists a bot as a verified bot, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the bot for a defined period of time.
+
+It takes 24 hours for an inactive IP to be removed as a verified bot.
+
+A bot can remain unlisted until Cloudflare sees traffic being sourced from the bot. When the bot is revalidated, it is listed as a verified bot again.

src/content/docs/bots/concepts/bot/verified-bots/policy.mdx

@@ -2,7 +2,7 @@
 pcx_content_type: reference
 title: Verified bots policy
 sidebar:
-    order: 2
+    order: 5
     label: Policy
 
 ---
@@ -27,7 +27,7 @@ A bot crawling one site is not valid.
 
 ### Bot Identification
 
-The user-agent with the following requirements:
+The user-agent or message signature with the following requirements:
 
 - Have at least 5 characters.
 - Must not contain special characters.
@@ -72,22 +72,6 @@ If a search engine crawler skips `robots.txt`, it will be rejected.
 
 The bot must have publicly documented expected behavior or user-agent format.
 
-## IP Validation
-
-A set of validation methods and requirements to gather set IP ranges for a verified service.
-
-### Public IP List
-
-- A fixed and limited set of IP addresses, which can be verified via publicly accessible plain-text, `JSON`, or `CSV`.
-- IP addresses used solely by the bot owner.
-- A user-agent match pattern.
-
-### Reverse DNS
-
-- A list of domain suffixes to validate DNS records.
-- IP addresses should have PTR records set correctly.
-- A user-agent match pattern.
-
 ## Breach of Policy
 
 If any of the requirements to validate are breached, a service will be removed from the global allowlist.
@@ -100,39 +84,3 @@ If any of the requirements to validate are breached, a service will be removed f
 - A block of IPs not briefed on onboarding is added to the list.
 - The disclosed purpose of the service does not reflect on the traffic.
 - An AI Crawler that does not respect the crawl-delay directive in robots.txt.
-
-## Online application
-
-To submit a verified bot that Cloudflare is not [currently tracking](https://radar.cloudflare.com/verified-bots), fill out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard for the fastest possible results. Bot operators who prefer not to create a free Cloudflare account can do so using our [old form](https://docs.google.com/forms/d/e/1FAIpQLSdqYNuULEypMnp4i5pROSc-uP6x65Xub9svD27mb8JChA_-XA/viewform?usp=sf_link), but the waiting time is up to several weeks for verified bot requests to be evaluated.
-
-### Generic user-agents
-
-User-agent patterns that match generic user-agents will be rejected by the Verified Bots API. When you add a user-agent pattern that is considered very common to the Verified Bot form, you may encounter an error message that will prompt you to correct the user-agent before you can submit again.
-
-Generic user-agents include:
-
-- `Dart`
-- `Go-http-client`
-- `GuzzleHttp`
-- `Google Chrome`
-- `Mozilla Firefox`
-- `Safari`
-- `Nessus`
-- `Websocket++`
-- `cloudflare-go`
-- `fasthttp`
-- `got`
-- `nginx-ssl early hints`
-- `node`
-- `node-fetch`
-- `okhttp`
-- `python-requests`
-- `uTorrent`
-
-## Transient false negatives
-
-Once Cloudflare lists a bot as a verified bot, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the bot for a defined period of time.
-
-It takes 24 hours for an inactive IP to be removed as a verified bot.
-
-A bot can remain unlisted until Cloudflare sees traffic being sourced from the bot. When the bot is revalidated, it is listed as a verified bot again.