From e01d3ac5d1a2a5e3b05d599daaf6d0ac3fbaa244 Mon Sep 17 00:00:00 2001 From: kodster28 Date: Mon, 9 Jun 2025 09:09:10 -0500 Subject: [PATCH 1/3] [Bots] Update AI bots info --- src/content/docs/bots/concepts/bot/index.mdx | 39 ++++++++++++++++--- .../partials/bots/ai-bots-definition.mdx | 38 ------------------ src/content/partials/bots/list-ai-bots.mdx | 30 ++++++++++++++ 3 files changed, 63 insertions(+), 44 deletions(-) delete mode 100644 src/content/partials/bots/ai-bots-definition.mdx create mode 100644 src/content/partials/bots/list-ai-bots.mdx diff --git a/src/content/docs/bots/concepts/bot/index.mdx b/src/content/docs/bots/concepts/bot/index.mdx index 4cfbcff4df02f45..d275037bd3f59b0 100644 --- a/src/content/docs/bots/concepts/bot/index.mdx +++ b/src/content/docs/bots/concepts/bot/index.mdx @@ -6,10 +6,9 @@ sidebar: learning_center: title: What is a bot? link: https://www.cloudflare.com/learning/bots/what-is-a-bot/ - --- -import { Render } from "~/components" +import { Render } from "~/components"; @@ -17,7 +16,7 @@ Bots can be used for good (chatbots, search engine crawlers) or for evil (invent :::note[More information] -For more background, refer to [What is a bot?](https://www.cloudflare.com/learning/bots/what-is-a-bot/). +For more background, refer to [What is a bot?](https://www.cloudflare.com/learning/bots/what-is-a-bot/). ::: ## Verified bots @@ -26,14 +25,42 @@ For more background, refer to [What is a bot?](https://www.cloudflare.com/learni :::note -The method for allowing or blocking verified bots depends on [your plan](/bots/get-started/). +The method for allowing or blocking verified bots depends on [your plan](/bots/get-started/). ::: ## AI bots - +You can block artificial intelligence (AI) bots, crawlers, and scrapers from scraping your website content and training large language models (LLM) to recreate it without your permission. + +### Which bots are blocked + + + +### How it works + +When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with `robots.txt` and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules. + +The rule to block AI bots takes precedence over all other Super Bot Fight Mode rules. For example, if you have enabled **Block AI bots** and **Allow verified bots**, verified AI bots will also be blocked even if you allow other verified bots on your website or application. + +For Bot Management customers, if you have set a rule to serve managed challenges to definitely automated bots, AI bots will also be challenged because custom rules run in a phase before Super Bot Fight Mode, which is the phase when the rule to block AI bots runs. + +This behavior remains the same if the setting for verified, definitely automated, and likely bots is set to `block` or `allow`. If you have an action to `allow` for these rules, the request is not matched to any rule and proceeds to the next ruleset phase. Similarly, if the action is set to `block`, they will be blocked in the earlier phase and do not move on to match the AI rule at all. However, when the action is `challenge`, the request matches a rule and therefore will not be matched to any rules after. + +For self-serve non-Bot Management customers, all rules for verified, definitely automated, and likely bots run in the phase following the AI bots rule. + +```mermaid +--- +title: Rule phases +--- +flowchart LR +accTitle: AI bots rule phases diagram +accDescr: This diagram details the phases in which AI bots rules run. +A[Custom rules] --> B[Block AI bots
managed rule] --> C[Allow verified bots rule] +``` + +This feature is available on all Cloudflare plans. :::note -The method for blocking AI bots depends on [your plan](/bots/get-started/). +The method for blocking AI bots depends on [your plan](/bots/get-started/). ::: diff --git a/src/content/partials/bots/ai-bots-definition.mdx b/src/content/partials/bots/ai-bots-definition.mdx deleted file mode 100644 index 4ac3d1f2c3fa1d6..000000000000000 --- a/src/content/partials/bots/ai-bots-definition.mdx +++ /dev/null @@ -1,38 +0,0 @@ ---- -{} - ---- - -You can block artificial intelligence (AI) bots, crawlers, and scrapers from scraping your website content and training large language models (LLM) to recreate it without your permission. - -When you enable this feature, all verified bots that are classified as AI Search, AI Assistant, AI Crawler, or an Archiver, as well as a number of unverified bots that fall into the [verified bot categories](/bots/concepts/bot/verified-bots/categories/) are blocked. It does not block verified bots that fall into the Search Engine categories. - -Refer to [Cloudflare Radar](https://radar.cloudflare.com/traffic/verified-bots) for examples of verified bots. - -:::note -Some AI bots overlap with definitely automated bots and verified bots, the latter becoming verified AI bots. - -For a partial list of verified AI Bots, refer to the [Cloudflare Radar](/radar/) categories of AI Search, AI Assistant, or AI Crawler, as well as some other bots that harvest data for AI training. -::: - -When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with `robots.txt` and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules. - -The rule to block AI bots takes precedence over all other Super Bot Fight Mode rules. For example, if you have enabled **Block AI bots** and **Allow verified bots**, verified AI bots will also be blocked even if you allow other verified bots on your website or application. - -For Bot Management customers, if you have set a rule to serve managed challenges to definitely automated bots, AI bots will also be challenged because custom rules run in a phase before Super Bot Fight Mode, which is the phase when the rule to block AI bots runs. - -This behavior remains the same if the setting for verified, definitely automated, and likely bots is set to `block` or `allow`. If you have an action to `allow` for these rules, the request is not matched to any rule and proceeds to the next ruleset phase. Similarly, if the action is set to `block`, they will be blocked in the earlier phase and do not move on to match the AI rule at all. However, when the action is `challenge`, the request matches a rule and therefore will not be matched to any rules after. - -For self-serve non-Bot Management customers, all rules for verified, definitely automated, and likely bots run in the phase following the AI bots rule. - -```mermaid ---- -title: Rule phases ---- -flowchart LR -accTitle: AI bots rule phases diagram -accDescr: This diagram details the phases in which AI bots rules run. -A[Custom rules] --> B[Block AI bots
managed rule] --> C[Allow verified bots rule] -``` - -This feature is available on all Cloudflare plans. diff --git a/src/content/partials/bots/list-ai-bots.mdx b/src/content/partials/bots/list-ai-bots.mdx new file mode 100644 index 000000000000000..d8a640658761362 --- /dev/null +++ b/src/content/partials/bots/list-ai-bots.mdx @@ -0,0 +1,30 @@ +--- +{} +--- + +When you enable this feature, Cloudflare will block the following bots: + +- `Amazonbot` (Amazon) +- `Applebot` (Apple) +- `Bytespider` (ByteDance) +- `ChatGPT-User` (OpenAI) +- `ClaudeBot` (Anthropic) +- `Claude-SearchBot` (Anthropic) +- `Claude-User` (Anthropic) +- `DuckAssistBot` (DuckDuckGo) +- `Google-CloudVertexBot` (Google) +- `GoogleOther` (Google) +- `GPTBot` (OpenAI) +- `Meta-ExternalAgent` (Meta) +- `OAI-SearchBot` (OpenAI) +- `PerplexityBot` (Perplexity) +- `PetalBot` (Huawei) +- `TikTokSpider` (ByteDance) + +In addition to this list, all verified bots that are classified as `AI Search`, `AI Assistant`, `AI Crawler`, or an `Archiver`, as well as a number of unverified bots that fall into the [verified bot categories](/bots/concepts/bot/verified-bots/categories/) are blocked. It does not block verified bots that fall into the `Search Engine` categories. + +:::note +Some AI bots overlap with definitely automated bots and verified bots, the latter becoming verified AI bots. + +For a partial list of verified AI Bots, refer to the [Cloudflare Radar](https://radar.cloudflare.com/bots#verified-bots) categories of AI Search, AI Assistant, or AI Crawler, as well as some other bots that harvest data for AI training. +::: From 24db84a611be47e26dc5bc1448bf6d60db3f03fd Mon Sep 17 00:00:00 2001 From: kodster28 Date: Mon, 9 Jun 2025 09:16:41 -0500 Subject: [PATCH 2/3] one more perplexity --- src/content/partials/bots/list-ai-bots.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/src/content/partials/bots/list-ai-bots.mdx b/src/content/partials/bots/list-ai-bots.mdx index d8a640658761362..0c47687436decc9 100644 --- a/src/content/partials/bots/list-ai-bots.mdx +++ b/src/content/partials/bots/list-ai-bots.mdx @@ -18,6 +18,7 @@ When you enable this feature, Cloudflare will block the following bots: - `Meta-ExternalAgent` (Meta) - `OAI-SearchBot` (OpenAI) - `PerplexityBot` (Perplexity) +- `Perplexity-User` (Perplexity) - `PetalBot` (Huawei) - `TikTokSpider` (ByteDance) From ea9854c638ef800b9db1918191c13289cde653a1 Mon Sep 17 00:00:00 2001 From: kodster28 Date: Mon, 9 Jun 2025 13:51:11 -0500 Subject: [PATCH 3/3] update to include ai audit --- src/content/docs/ai-audit/get-started.mdx | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/src/content/docs/ai-audit/get-started.mdx b/src/content/docs/ai-audit/get-started.mdx index 70d9b364d776b63..f48e08e0768778c 100644 --- a/src/content/docs/ai-audit/get-started.mdx +++ b/src/content/docs/ai-audit/get-started.mdx @@ -11,7 +11,7 @@ head: description: Learn how to set up AI Audit. --- -import { Render, Steps } from "~/components"; +import { Details, Render, Steps } from "~/components"; This guide instructs you through @@ -36,6 +36,16 @@ To use AI Audit: 4. From the **Bot traffic** page, under **Block AI Bots**, select **Enable**. +
+ +
+ +:::note + +For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works). + +::: + ## 2. Block specific bot categories (Enterprise plan only) Customers on the Enterprise plan -- and with a [Bot Management subscription](/bots/plans/bm-subscription/) -- can choose to only block specific AI crawlers, while allowing others.