Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions public/__redirects
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,9 @@
/bots/get-started/pro/ /bots/get-started/super-bot-fight-mode/ 301
/bots/additional-configurations/javascript-detections/ /cloudflare-challenges/challenge-types/javascript-detections/ 301
/bots/troubleshooting/frequently-asked-questions/ /bots/frequently-asked-questions/ 301
/bots/concepts/bot/verified-bots/categories/ /bots/concepts/bot/verified-bots/#categories 301
/bots/concepts/bot/verified-bots/ip-validation/ /bots/reference/bot-verification/ip-validation/ 301
/bots/concepts/bot/verified-bots/web-bot-auth/ /bots/reference/bot-verification/web-bot-auth/ 301

#browser-rendering
/browser-rendering/get-started/browser-rendering-with-DO/ /browser-rendering/workers-bindings/browser-rendering-with-do/ 301
Expand Down
4 changes: 2 additions & 2 deletions src/content/docs/ai-audit/features/manage-ai-crawlers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ The **AI Crawlers** tab displays a table of AI crawlers that are requesting acce
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| AI Crawlers | The name of the AI crawler. |
| Operator | The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers. |
| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/). |
| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/#categories). |
| Block | Toggle for blocking specific AI crawlers. |

</TabItem> <TabItem label="Paid plan">
| Column | Details |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| AI Crawlers | The name of the AI crawler. |
| Operator | The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers. |
| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/). |
| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/#categories). |
| Requests | The number of times the AI crawler has requested to crawl your content. |
| Robots.txt violations | The number of times the AI crawler has violated your <GlossaryTooltip term="robots.txt">`robots.txt`</GlossaryTooltip> file. |
| Action | The action you wish to take for the AI crawler. Refer to [Take action for each AI crawler](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ To access this content, the AI crawler must provide headers for paid access.

### 2.1. Include WBA headers

Include Web Bot Auth headers by following the steps [Sign your requests](/bots/concepts/bot/verified-bots/web-bot-auth/#4-after-verification-sign-your-requests)
Include Web Bot Auth headers by following the steps [Sign your requests](/bots/reference/bot-verification/web-bot-auth/#4-after-verification-sign-your-requests)

### 2.2. Include payment headers

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ To access this content, you must verify your AI crawler.

Ensure your AI crawler identifies itself with the required headers for Web Bot Auth.

Follow the steps found in [Web Both Auth](/bots/concepts/bot/verified-bots/web-bot-auth/).
Follow the steps found in [Web Both Auth](/bots/reference/bot-verification/web-bot-auth/).

## 2. Follow verified bot policy

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import { Render, Steps } from "~/components"
The **Block AI bots** feature is only available in the new [application security dashboard](/security/).
:::

You can choose to block AI bots by activating **Block AI bots**. Activating this setting will block [verified bots](/bots/concepts/bot/verified-bots/overview/) that are classified as AI crawlers, as well as a number of unverified bots that behave similarly.
You can choose to block AI bots by activating **Block AI bots**. Activating this setting will block [verified bots](/bots/concepts/bot/verified-bots/) that are classified as AI crawlers, as well as a number of unverified bots that behave similarly.

To block [AI bots](/bots/concepts/bot/#ai-bots):

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/bots/concepts/bot/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ For more background, refer to [What is a bot?](https://www.cloudflare.com/learni

<Render file="verified-bots" product="bots" />

For more information, refer to [Verified bots](/bots/concepts/bot/verified-bots/overview/).
For more information, refer to [Verified bots](/bots/concepts/bot/verified-bots/).

:::note

Expand Down
41 changes: 41 additions & 0 deletions src/content/docs/bots/concepts/bot/signed-agents/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
pcx_content_type: overview
title: Signed agents
sidebar:
order: 3
learning_center:
title: What is a bot?
link: https://www.cloudflare.com/learning/bots/what-is-a-bot/

---

A signed agent is controlled by an end user and a verified signature-agent from their Web Bot Auth implementation.

You can request for your agent to be added to Cloudflare's bots and agents directory by filling out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard.

:::note
A bot cannot be registered as both a verified bot and a signed agent. Review Cloudflare's [verified bots](/bots/concepts/bot/verified-bots/) to determine how to identify your bot.
:::

## Signed agent requirement

For an agent to be recognized, it must meet the following requirements:

1. The agent must follow the [signed agents policy](/bots/concepts/bot/signed-agents/policy/).
2. The bot must be using [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/).

Once Cloudflare approves a signed agent, it should appear on [Cloudflare Radar's bots and agents directory](https://radar.cloudflare.com/verified-bots).

---

## Transient false negatives

Once Cloudflare lists an agent as a signed agent, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the agent for a defined period of time.

An agent can remain unlisted until Cloudflare sees traffic being sourced from the agent. When the agent is revalidated, it is listed as a signed agent again.

---

## Verification method

The bot must be verified using [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/).
68 changes: 68 additions & 0 deletions src/content/docs/bots/concepts/bot/signed-agents/policy.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
pcx_content_type: reference
title: Signed agents policy
sidebar:
order: 3
label: Policy

---

In order to be listed by Cloudflare as a signed agent, your agent must conform to the below requirements. To provide the best possible protection to our customers, this policy may change in the future as we adapt to new bot behaviors.

## Agent policy

### Minimum traffic

An agent must have a minimum amount of traffic for Cloudflare to be able to find it in the sampled data. The minimum traffic should have more than 1,000 requests per day across multiple domains.

### Minimum zones

Service must be made for a widespread use of zones.

#### Example

A bot crawling one site is not valid.

### Agent identification

The user-agent field is optional as it is not required for Web Bot Authentication.

However, if you choose to provide a user-agent, it and the message signature must meet the following requirements:

- Have at least five characters.
- Must not contain special characters.
- Must not include the same user-agent of another verified service.

#### Example

`cloudflare-browser-rendering` is a valid message signature.

### Service purpose

The purpose of the service should be benign or helpful to both the owner of a zone and the users of the service. The service cannot perform any of the following:

- Bot tooling
- Scalpers
- Credential-stuffing
- Directory-traversal scanning
- Excessive data scraping
- DDoS botnets

#### Example

Price scraping direct e-commerce competitors is not a valid use case.

### Public documentation

The agent must have a publicly documented purpose and expected behavior.

---

## Breach of policy

If any of the requirements to validate are breached, a service will be removed from the signed agent list.

The following are examples of breaches of policy:

- The service has vulnerabilities that have not been patched.
- The disclosed purpose of the service does not reflect on the traffic.
41 changes: 0 additions & 41 deletions src/content/docs/bots/concepts/bot/verified-bots/categories.mdx

This file was deleted.

78 changes: 71 additions & 7 deletions src/content/docs/bots/concepts/bot/verified-bots/index.mdx
Original file line number Diff line number Diff line change
@@ -1,20 +1,84 @@
---
pcx_content_type: navigation
pcx_content_type: overview
title: Verified bots
sidebar:
group:
hideIndex: true
order: 2
learning_center:
title: What is a bot?
link: https://www.cloudflare.com/learning/bots/what-is-a-bot/

---

import { Render, DirectoryListing } from "~/components";
import { GlossaryTooltip } from "~/components"

<Render file="verified-bots" product="bots" />
A verified bot is a bot which has been added to Cloudflare's list of <GlossaryTooltip term="verified bot">verified bots</GlossaryTooltip>.

Refer to the pages below for additional resources on verified bots.
You can request for your bot to be added to Cloudflare's bots and agents directory by filling out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard.

<DirectoryListing />
:::note
A bot cannot be registered as both a verified bot and a signed agent. Review Cloudflare's [signed agents](/bots/concepts/bot/signed-agents/) to determine how to identify your bot.
:::

## Verified bot requirement

For a bot to be verified, it must meet the following requirements:

1. The bot must follow [verified bots policy](/bots/concepts/bot/verified-bots/policy/).
2. The bot must be verified using one of the following verification methods:
- [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/)
- [IP validation](/bots/reference/bot-verification/ip-validation/)

Once Cloudflare approves a verified bot, it should appear on [Cloudflare Radar's bots and agents directory](https://radar.cloudflare.com/verified-bots).

---

## Transient false negatives

Once Cloudflare lists a bot as a verified bot, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the bot for a defined period of time.

It takes 24 hours for an inactive IP to be removed as a verified bot.

A bot can remain unlisted until Cloudflare sees traffic being sourced from the bot. When the bot is revalidated, it is listed as a verified bot again.

---

## Verification methods

The bot must be verified using one of the following validation methods:

- [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/)
- [IP validation](/bots/reference/bot-verification/ip-validation/)

---

## Categories

You can segment your verified bot traffic by its type and purpose by adding the Verified Bot Categories field `cf.verified_bot_category` as a filter criteria in [WAF Custom rules](/waf/custom-rules/), [Advanced Rate Limiting](/waf/rate-limiting-rules/), and Late Transform rules.

:::note
The Verified Bot Categories field is not compatible with legacy Firewall rules.
:::

| Name | String value | Example |
| -------------------------- | ---------------------------- | ----------------------------------------------------------------------- |
| Academic research | `Academic Research` | Library of Congress, TurnItInBot, Bibliothèque nationale de France |
| Accessibility | `Accessibility` | Accessible Web Bot |
| Advertising or marketing | `Advertising & Marketing` | Google Adsbot |
| Aggregators | `Aggregator` | Pinterest, Indeed Jobsbot |
| AI Assistant | `AI Assistant` | Perplexity-User, DuckAssistBot |
| AI Crawler | `AI Crawler` | Google Bard, ChatGPT bot |
| AI Search | `AI Search` | OAI-SearchBot |
| Archiver | `Archiver` | Internet Archive, CommonCrawl |
| Feed fetcher | `Feed Fetcher` | RSS or Podcast feed updaters |
| Monitoring or analytics | `Monitoring & Analytics` | Uptime Monitors |
| Page preview | `Page Preview` | Facebook, Slack, Twitter, or Discord Link Preview tools |
| Search engine crawler | `Search Engine Crawler` | Googlebot, Bingbot, Yandexbot, Baidubot |
| Search engine optimization | `Search Engine Optimization` | Google Lighthouse, GT Metrix, Pingdom, AddThis |
| Security | `Security` | Vulnerability Scanners, SSL Domain Control Validation (DCV) Check Tools |
| Social media marketing | `Social Media Marketing` | Brandwatch |
| Webhooks | `Webhooks` | Payment processors, WordPress Integration tools |
| Other | `Other` | |

### Availability

Verified Bot Categories is available on all plans.
37 changes: 0 additions & 37 deletions src/content/docs/bots/concepts/bot/verified-bots/overview.mdx

This file was deleted.

Loading
Loading