Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/content/docs/bots/concepts/bot/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ The method for allowing or blocking verified bots depends on [your plan](/bots/g

## AI bots

You can block artificial intelligence (AI) bots, crawlers, and scrapers from scraping your website content and training large language models (LLM) to recreate it without your permission.
You can opt into a managed rule that will block bots that we categorize as artificial intelligence (AI) crawlers (“AI Bots”) from visiting your website. Customers may choose to do this to prevent AI-related usage of their content, such as training large language models (LLM).

### Which bots are blocked

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,15 @@
pcx_content_type: reference
title: Verified bot categories
sidebar:
order: 3
label: Categories

order: 3
label: Categories
---

You can segment your verified bot traffic by its type and purpose by adding the Verified Bot Categories field `cf.verified_bot_category` as a filter criteria in [WAF Custom rules](/waf/custom-rules/), [Advanced Rate Limiting](/waf/rate-limiting-rules/), and Late Transform rules.

:::note

The Verified Bot Categories field is not compatible with legacy Firewall rules.
The Verified Bot Categories field is not compatible with legacy Firewall rules.
:::

## Categories
Expand All @@ -24,8 +23,8 @@ The Verified Bot Categories field is not compatible with legacy Firewall rules.
| Aggregators | `Aggregator` | Pinterest, Indeed Jobsbot |
| AI Assistant | `AI Assistant` | Perplexity-User, DuckAssistBot |
| AI Crawler | `AI Crawler` | Google Bard, ChatGPT bot |
| AI Search | `AI Search` | SearchGPT |
| Archiver | `Archiver` | Internet Archive, CommonCrawl |
| AI Search | `AI Search` | OAI-SearchBot |
| Archiver | `Archiver` | Internet Archive, CommonCrawl |
| Feed fetcher | `Feed Fetcher` | RSS or Podcast feed updaters |
| Monitoring or analytics | `Monitoring & Analytics` | Uptime Monitors |
| Page preview | `Page Preview` | Facebook, Slack, Twitter, or Discord Link Preview tools |
Expand Down
20 changes: 5 additions & 15 deletions src/content/partials/bots/list-ai-bots.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,34 +2,24 @@
{}
---

When you enable this feature, Cloudflare will block bots that have been identified as AI Crawlers. These bots include, but are not limited to, the following bots:
When you enable this feature, Cloudflare will block the following bots:

- `Amazonbot` (Amazon)
- `Applebot` (Apple)
- `Bytespider` (ByteDance)
- `ChatGPT-User` (OpenAI)
- `ClaudeBot` (Anthropic)
- `Claude-SearchBot` (Anthropic)
- `Claude-User` (Anthropic)
- `DuckAssistBot` (DuckDuckGo)
- `Google-CloudVertexBot` (Google)
- `GoogleOther` (Google)
- `GPTBot` (OpenAI)
- `Meta-ExternalAgent` (Meta)
- `OAI-SearchBot` (OpenAI)
- `PerplexityBot` (Perplexity)
- `Perplexity-User` (Perplexity)
- `PetalBot` (Huawei)
- `TikTokSpider` (ByteDance)
- `CCBot` (Common Crawl)

In addition to this list, all verified bots that are classified as `AI Search`, `AI Assistant`, `AI Crawler`, or an `Archiver`, as well as a number of unverified bots that fall into the [verified bot categories](/bots/concepts/bot/verified-bots/categories/) are blocked. It does not block verified bots that fall into the `Search Engine` categories.
In addition to this list, [verified bots](https://radar.cloudflare.com/bots#verified-bots) that are classified in AI-related categories such as `AI Assistant`, `AI Crawler`, or an `Archiver`, as well as a number of unverified bots that behave similarly, are included in the rule. This rule does not include verified bots that fall into the `Search Engine` categories.

The bots classified in these categories may change in real time as new AI bots are detected.
These categories, and the bots classified in these categories, may change from time to time.

:::note
Some AI bots overlap with definitely automated bots and verified bots, the latter becoming verified AI bots.

For a partial list of verified AI Bots, refer to the [Cloudflare Radar](https://radar.cloudflare.com/bots#verified-bots) categories of AI Search, AI Assistant, or AI Crawler, as well as some other bots that harvest data for AI training.
:::

If you are a bot operator and feel your bot may have been incorrectly categorized, [add your bot to the list of verified bots](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots).
If you are a bot operator and feel your bot may have been incorrectly categorized, [add your bot to the list of verified bots](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots).