[Bots] Signed agents (#24757)

* signed agents

* spacing

* redirect

* space

* fix link

* broken links

Author: patriciasantaana

public/__redirects

@@ -265,6 +265,10 @@
 /bots/get-started/pro/ /bots/get-started/super-bot-fight-mode/ 301
 /bots/additional-configurations/javascript-detections/ /cloudflare-challenges/challenge-types/javascript-detections/ 301
 /bots/troubleshooting/frequently-asked-questions/ /bots/frequently-asked-questions/ 301
+/bots/concepts/bot/verified-bots/categories/ /bots/concepts/bot/verified-bots/#categories 301
+/bots/concepts/bot/verified-bots/ip-validation/ /bots/reference/bot-verification/ip-validation/ 301
+/bots/concepts/bot/verified-bots/web-bot-auth/ /bots/reference/bot-verification/web-bot-auth/ 301
+/bots/concepts/bot/verified-bots/overview/  /bots/concepts/bot/verified-bots/ 301
 
 #browser-rendering
 /browser-rendering/get-started/browser-rendering-with-DO/ /browser-rendering/workers-bindings/browser-rendering-with-do/ 301

src/content/docs/ai-crawl-control/features/manage-ai-crawlers.mdx

@@ -22,7 +22,7 @@ The **Crawlers** tab displays a table of AI crawlers that are requesting access
 | Column                | Details                                                                                                                                                                     |
 | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | Crawler               | The name of the AI crawler and the operator that owns it.                                                                                                                   |
-| Category              | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/).                                                           |
+| Category              | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/#categories).                                                           |
 | Requests              | Total allowed and blocked requests with trend chart. Blocked requests may come from any configured rule, not just the actions shown here.                                   |
 | Robots.txt violations | The number of times the AI crawler has violated your <GlossaryTooltip term="robots.txt">`robots.txt`</GlossaryTooltip> file.                                                |
 | Action                | The action you wish to take for the AI crawler. Refer to [Take action for each AI crawler](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). |

src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx

@@ -33,7 +33,7 @@ To access this content, the AI crawler must provide headers for paid access.
 
 ### 2.1. Include WBA headers
 
-Include Web Bot Auth headers by following the steps [Sign your requests](/bots/concepts/bot/verified-bots/web-bot-auth/#4-after-verification-sign-your-requests)
+Include Web Bot Auth headers by following the steps [Sign your requests](/bots/reference/bot-verification/web-bot-auth/#4-after-verification-sign-your-requests)
 
 ### 2.2. Include payment headers
 

src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/verify-ai-crawler.mdx

@@ -37,7 +37,7 @@ To access this content, you must verify your AI crawler.
 
 Ensure your AI crawler identifies itself with the required headers for Web Bot Auth.
 
-Follow the steps found in [Web Both Auth](/bots/concepts/bot/verified-bots/web-bot-auth/).
+Follow the steps found in [Web Both Auth](/bots/reference/bot-verification/web-bot-auth/).
 
 ## 2. Follow verified bot policy
 

src/content/docs/bots/additional-configurations/block-ai-bots.mdx

@@ -14,7 +14,7 @@ import { Render, Steps } from "~/components"
 The **Block AI bots** feature is only available in the new [application security dashboard](/security/).
 :::
 
-You can choose to block AI bots by activating **Block AI bots**. Activating this setting will block [verified bots](/bots/concepts/bot/verified-bots/overview/) that are classified as AI crawlers, as well as a number of unverified bots that behave similarly.
+You can choose to block AI bots by activating **Block AI bots**. Activating this setting will block [verified bots](/bots/concepts/bot/verified-bots/) that are classified as AI crawlers, as well as a number of unverified bots that behave similarly.
 
 To block [AI bots](/bots/concepts/bot/#ai-bots):
 

src/content/docs/bots/concepts/bot/index.mdx

@@ -23,7 +23,7 @@ For more background, refer to [What is a bot?](https://www.cloudflare.com/learni
 
 <Render file="verified-bots" product="bots" />
 
-For more information, refer to [Verified bots](/bots/concepts/bot/verified-bots/overview/).
+For more information, refer to [Verified bots](/bots/concepts/bot/verified-bots/).
 
 :::note
 

src/content/docs/bots/concepts/bot/signed-agents/index.mdx

@@ -0,0 +1,41 @@
+---
+pcx_content_type: overview
+title: Signed agents
+sidebar:
+  order: 3
+learning_center:
+  title: What is a bot?
+  link: https://www.cloudflare.com/learning/bots/what-is-a-bot/
+
+---
+
+A signed agent is controlled by an end user and a verified signature-agent from their Web Bot Auth implementation.
+
+You can request for your agent to be added to Cloudflare's bots and agents directory by filling out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard.
+
+:::note
+A bot cannot be registered as both a verified bot and a signed agent. Review Cloudflare's [verified bots](/bots/concepts/bot/verified-bots/) to determine how to identify your bot.
+:::
+
+## Signed agent requirement
+
+For an agent to be recognized, it must meet the following requirements:
+
+1. The agent must follow the [signed agents policy](/bots/concepts/bot/signed-agents/policy/).
+2. The bot must be using [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/).
+
+Once Cloudflare approves a signed agent, it should appear on [Cloudflare Radar's bots and agents directory](https://radar.cloudflare.com/verified-bots).
+
+---
+
+## Transient false negatives
+
+Once Cloudflare lists an agent as a signed agent, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the agent for a defined period of time.
+
+An agent can remain unlisted until Cloudflare sees traffic being sourced from the agent. When the agent is revalidated, it is listed as a signed agent again.
+
+---
+
+## Verification method
+
+The bot must be verified using [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/).

src/content/docs/bots/concepts/bot/signed-agents/policy.mdx

@@ -0,0 +1,68 @@
+---
+pcx_content_type: reference
+title: Signed agents policy
+sidebar:
+    order: 3
+    label: Policy
+
+---
+
+In order to be listed by Cloudflare as a signed agent, your agent must conform to the below requirements. To provide the best possible protection to our customers, this policy may change in the future as we adapt to new bot behaviors.
+
+## Agent policy
+
+### Minimum traffic
+
+An agent must have a minimum amount of traffic for Cloudflare to be able to find it in the sampled data. The minimum traffic should have more than 1,000 requests per day across multiple domains.
+
+### Minimum zones
+
+Service must be made for a widespread use of zones.
+
+#### Example
+
+A bot crawling one site is not valid.
+
+### Agent identification
+
+The user-agent field is optional as it is not required for Web Bot Authentication.
+
+However, if you choose to provide a user-agent, it and the message signature must meet the following requirements:
+
+- Have at least five characters.
+- Must not contain special characters.
+- Must not include the same user-agent of another verified service.
+
+#### Example
+
+`cloudflare-browser-rendering` is a valid message signature.
+
+### Service purpose
+
+The purpose of the service should be benign or helpful to both the owner of a zone and the users of the service. The service cannot perform any of the following:
+
+- Bot tooling
+- Scalpers
+- Credential-stuffing
+- Directory-traversal scanning
+- Excessive data scraping
+- DDoS botnets
+
+#### Example
+
+Price scraping direct e-commerce competitors is not a valid use case.
+
+### Public documentation
+
+The agent must have a publicly documented purpose and expected behavior.
+
+---
+
+## Breach of policy
+
+If any of the requirements to validate are breached, a service will be removed from the signed agent list.
+
+The following are examples of breaches of policy:
+
+- The service has vulnerabilities that have not been patched.
+- The disclosed purpose of the service does not reflect on the traffic.

src/content/docs/bots/concepts/bot/verified-bots/categories.mdx

@@ -1,20 +1,84 @@
 ---
-pcx_content_type: navigation
+pcx_content_type: overview
 title: Verified bots
 sidebar:
-  group:
-    hideIndex: true
   order: 2
 learning_center:
   title: What is a bot?
   link: https://www.cloudflare.com/learning/bots/what-is-a-bot/
 
 ---
 
-import { Render, DirectoryListing } from "~/components";
+import { GlossaryTooltip } from "~/components";
 
-<Render file="verified-bots" product="bots" />
+A verified bot is a bot which has been added to Cloudflare's list of <GlossaryTooltip term="verified bot">verified bots</GlossaryTooltip>.
 
-Refer to the pages below for additional resources on verified bots.
+You can request for your bot to be added to Cloudflare's bots and agents directory by filling out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard.
 
-<DirectoryListing />
+:::note
+A bot cannot be registered as both a verified bot and a signed agent. Review Cloudflare's [signed agents](/bots/concepts/bot/signed-agents/) to determine how to identify your bot.
+:::
+
+## Verified bot requirement
+
+For a bot to be verified, it must meet the following requirements:
+
+1. The bot must follow [verified bots policy](/bots/concepts/bot/verified-bots/policy/).
+2. The bot must be verified using one of the following verification methods:
+   - [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/)
+   - [IP validation](/bots/reference/bot-verification/ip-validation/)
+
+Once Cloudflare approves a verified bot, it should appear on [Cloudflare Radar's bots and agents directory](https://radar.cloudflare.com/verified-bots).
+
+---
+
+## Transient false negatives
+
+Once Cloudflare lists a bot as a verified bot, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the bot for a defined period of time.
+
+It takes 24 hours for an inactive IP to be removed as a verified bot.
+
+A bot can remain unlisted until Cloudflare sees traffic being sourced from the bot. When the bot is revalidated, it is listed as a verified bot again.
+
+---
+
+## Verification methods
+
+The bot must be verified using one of the following validation methods:
+
+- [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/)
+- [IP validation](/bots/reference/bot-verification/ip-validation/)
+
+---
+
+## Categories
+
+You can segment your verified bot traffic by its type and purpose by adding the Verified Bot Categories field `cf.verified_bot_category` as a filter criteria in [WAF Custom rules](/waf/custom-rules/), [Advanced Rate Limiting](/waf/rate-limiting-rules/), and Late Transform rules.
+
+:::note
+The Verified Bot Categories field is not compatible with legacy Firewall rules.
+:::
+
+| Name                       | String value                 | Example                                                                 |
+| -------------------------- | ---------------------------- | ----------------------------------------------------------------------- |
+| Academic research          | `Academic Research`          | Library of Congress, TurnItInBot, Bibliothèque nationale de France      |
+| Accessibility              | `Accessibility`              | Accessible Web Bot                                                      |
+| Advertising or marketing   | `Advertising & Marketing`    | Google Adsbot                                                           |
+| Aggregators                | `Aggregator`                 | Pinterest, Indeed Jobsbot                                               |
+| AI Assistant               | `AI Assistant`               | Perplexity-User, DuckAssistBot                                          |
+| AI Crawler                 | `AI Crawler`                 | Google Bard, ChatGPT bot                                                |
+| AI Search                  | `AI Search`                  | OAI-SearchBot                                                           |
+| Archiver                   | `Archiver`                   | Internet Archive, CommonCrawl                                           |
+| Feed fetcher               | `Feed Fetcher`               | RSS or Podcast feed updaters                                            |
+| Monitoring or analytics    | `Monitoring & Analytics`     | Uptime Monitors                                                         |
+| Page preview               | `Page Preview`               | Facebook, Slack, Twitter, or Discord Link Preview tools                 |
+| Search engine crawler      | `Search Engine Crawler`      | Googlebot, Bingbot, Yandexbot, Baidubot                                 |
+| Search engine optimization | `Search Engine Optimization` | Google Lighthouse, GT Metrix, Pingdom, AddThis                          |
+| Security                   | `Security`                   | Vulnerability Scanners, SSL Domain Control Validation (DCV) Check Tools |
+| Social media marketing     | `Social Media Marketing`     | Brandwatch                                                              |
+| Webhooks                   | `Webhooks`                   | Payment processors, WordPress Integration tools                         |
+| Other                      | `Other`                      |                                                                         |
+
+### Availability
+
+Verified Bot Categories is available on all plans.