From 08dd85f92e78be9351c5db41451d96fe28ed48f2 Mon Sep 17 00:00:00 2001 From: Patricia Loraine Santa Ana Date: Mon, 28 Oct 2024 10:55:40 -0700 Subject: [PATCH 1/5] block ai bots updates --- .../partials/bots/ai-bots-definition.mdx | 27 ++++++++++++++++++- .../partials/bots/block-ai-bots-enable.mdx | 4 ++- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/src/content/partials/bots/ai-bots-definition.mdx b/src/content/partials/bots/ai-bots-definition.mdx index f31773ea5169657..010ea8e68011324 100644 --- a/src/content/partials/bots/ai-bots-definition.mdx +++ b/src/content/partials/bots/ai-bots-definition.mdx @@ -3,6 +3,31 @@ --- -You can block artificial intelligence (AI) bots, crawlers, and scrapers from scraping your website content and training large language models (LLM) to recreate it without your permission. When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that respect `robots.txt` and crawl rate, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules. +You can block artificial intelligence (AI) bots, crawlers, and scrapers from scraping your website content and training large language models (LLM) to recreate it without your permission. + +:::note +Some AI bots overlap with definitely automated bots and verified bots, the latter becoming verified AI bots. +::: + +When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with `robots.txt` and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules. + +The rule to block AI bots takes precedence over all other Super Bot Fight Mode rules. For example, if you have enabled **Block AI bots** and **Allow verified bots**, verified AI bots will also be blocked even if you allow other verified bots on your website or application. + +For Bot Management customers, if you have set a rule to serve managed challenges to definitely automated bots, AI bots will also be challenged because custom rules run in a phase before Super Bot Fight Mode, which is the phase when the rule to block AI bots runs. + +This behavior remains the same if the setting for verified, definitely automated, and likely bots is set to `block` or `allow`. If you have an action to `allow` for these rules, the request is not matched to any rule and proceeds to the next ruleset phase. Similarly, if the action is set to block, they will be blocked in the earlier phase and do not move on to match the AI rule at all. However, when the action is `challenge`, the request matches a rule and therefore will not be matched to any rules after. + +For non-Bot Management customers, all rules for verified, definitely automated, and likely bots run in the phase following the AI bots rule. + +```mermaid +--- +title: Rule phases +--- +flowchart LR +accTitle: AI bots rule phases diagram +accDescr: This diagram details the phases in which AI bots rules run. +A[Custom rules] --> B[Block AI bots rule] --> C[Allow verified bots rule] + +``` This feature is available on all Cloudflare plans. diff --git a/src/content/partials/bots/block-ai-bots-enable.mdx b/src/content/partials/bots/block-ai-bots-enable.mdx index b998b5f3cd7de90..4900447ec053ef8 100644 --- a/src/content/partials/bots/block-ai-bots-enable.mdx +++ b/src/content/partials/bots/block-ai-bots-enable.mdx @@ -5,7 +5,9 @@ inputParameters: param1 import { Markdown } from "~/components" +To block AI bots: + 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain. 2. Go to **Security** > **Bots**. 3. Select **Configure {props.one}**. -4. Enable **Block AI bots**. +4. Enable **Block AI bots**. \ No newline at end of file From 123aeb143f7f7c789581a23283c55389c1035427 Mon Sep 17 00:00:00 2001 From: Patricia Loraine Santa Ana Date: Mon, 28 Oct 2024 10:57:05 -0700 Subject: [PATCH 2/5] space --- src/content/partials/bots/ai-bots-definition.mdx | 1 - 1 file changed, 1 deletion(-) diff --git a/src/content/partials/bots/ai-bots-definition.mdx b/src/content/partials/bots/ai-bots-definition.mdx index 010ea8e68011324..41db8c789453de0 100644 --- a/src/content/partials/bots/ai-bots-definition.mdx +++ b/src/content/partials/bots/ai-bots-definition.mdx @@ -27,7 +27,6 @@ flowchart LR accTitle: AI bots rule phases diagram accDescr: This diagram details the phases in which AI bots rules run. A[Custom rules] --> B[Block AI bots rule] --> C[Allow verified bots rule] - ``` This feature is available on all Cloudflare plans. From 7001e2a1c2df805ae845fa6a3391af3931586921 Mon Sep 17 00:00:00 2001 From: Patricia Loraine Santa Ana Date: Mon, 28 Oct 2024 13:23:00 -0700 Subject: [PATCH 3/5] update placement and link --- src/content/docs/bots/get-started/biz-and-ent.mdx | 2 -- src/content/docs/bots/get-started/bm-subscription.mdx | 2 -- src/content/docs/bots/get-started/free.mdx | 2 -- src/content/docs/bots/get-started/pro.mdx | 2 -- src/content/partials/bots/block-ai-bots-enable.mdx | 2 +- 5 files changed, 1 insertion(+), 9 deletions(-) diff --git a/src/content/docs/bots/get-started/biz-and-ent.mdx b/src/content/docs/bots/get-started/biz-and-ent.mdx index aebb7c66f54616e..02f6321284679e8 100644 --- a/src/content/docs/bots/get-started/biz-and-ent.mdx +++ b/src/content/docs/bots/get-started/biz-and-ent.mdx @@ -37,8 +37,6 @@ If you have an Enterprise subscription without Bot Management, your application ## Block AI bots - - :::note diff --git a/src/content/docs/bots/get-started/bm-subscription.mdx b/src/content/docs/bots/get-started/bm-subscription.mdx index 9f920b025f8cd31..b53eeb188b9e9d0 100644 --- a/src/content/docs/bots/get-started/bm-subscription.mdx +++ b/src/content/docs/bots/get-started/bm-subscription.mdx @@ -33,8 +33,6 @@ If you are not seeing Bot Management enabled on your zone or if you still see ** ## Block AI bots - - :::note diff --git a/src/content/docs/bots/get-started/free.mdx b/src/content/docs/bots/get-started/free.mdx index e37a20eabb8c8f2..af166d538d28ca7 100644 --- a/src/content/docs/bots/get-started/free.mdx +++ b/src/content/docs/bots/get-started/free.mdx @@ -29,8 +29,6 @@ To disable Bot Fight Mode: ## Block AI bots - - :::note diff --git a/src/content/docs/bots/get-started/pro.mdx b/src/content/docs/bots/get-started/pro.mdx index b38460e5dcf2fcd..f593f1b2ec8e49a 100644 --- a/src/content/docs/bots/get-started/pro.mdx +++ b/src/content/docs/bots/get-started/pro.mdx @@ -32,8 +32,6 @@ Super Bot Fight Mode is included in your Pro subscription. When enabled, the pro ## Block AI bots - - :::note diff --git a/src/content/partials/bots/block-ai-bots-enable.mdx b/src/content/partials/bots/block-ai-bots-enable.mdx index 4900447ec053ef8..32ce77677eae3ea 100644 --- a/src/content/partials/bots/block-ai-bots-enable.mdx +++ b/src/content/partials/bots/block-ai-bots-enable.mdx @@ -5,7 +5,7 @@ inputParameters: param1 import { Markdown } from "~/components" -To block AI bots: +To block [AI bots](/bots/concepts/bot/#ai-bots): 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain. 2. Go to **Security** > **Bots**. From 8a88742d34adb195d953131bfe237285018b4000 Mon Sep 17 00:00:00 2001 From: Patricia Santa Ana <103445940+patriciasantaana@users.noreply.github.com> Date: Tue, 29 Oct 2024 09:24:34 -0700 Subject: [PATCH 4/5] wording edit --- src/content/partials/bots/ai-bots-definition.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/partials/bots/ai-bots-definition.mdx b/src/content/partials/bots/ai-bots-definition.mdx index 41db8c789453de0..fbe6fac5694ebfd 100644 --- a/src/content/partials/bots/ai-bots-definition.mdx +++ b/src/content/partials/bots/ai-bots-definition.mdx @@ -17,7 +17,7 @@ For Bot Management customers, if you have set a rule to serve managed challenges This behavior remains the same if the setting for verified, definitely automated, and likely bots is set to `block` or `allow`. If you have an action to `allow` for these rules, the request is not matched to any rule and proceeds to the next ruleset phase. Similarly, if the action is set to block, they will be blocked in the earlier phase and do not move on to match the AI rule at all. However, when the action is `challenge`, the request matches a rule and therefore will not be matched to any rules after. -For non-Bot Management customers, all rules for verified, definitely automated, and likely bots run in the phase following the AI bots rule. +For self-serve non-Bot Management customers, all rules for verified, definitely automated, and likely bots run in the phase following the AI bots rule. ```mermaid --- From a70da2b76906337328c8391cd1a713f993a1392e Mon Sep 17 00:00:00 2001 From: Patricia Santa Ana <103445940+patriciasantaana@users.noreply.github.com> Date: Tue, 29 Oct 2024 11:29:33 -0700 Subject: [PATCH 5/5] Apply suggestions from code review Co-authored-by: Pedro Sousa <680496+pedrosousa@users.noreply.github.com> --- src/content/partials/bots/ai-bots-definition.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/content/partials/bots/ai-bots-definition.mdx b/src/content/partials/bots/ai-bots-definition.mdx index fbe6fac5694ebfd..d0ad7058e563627 100644 --- a/src/content/partials/bots/ai-bots-definition.mdx +++ b/src/content/partials/bots/ai-bots-definition.mdx @@ -15,7 +15,7 @@ The rule to block AI bots takes precedence over all other Super Bot Fight Mode r For Bot Management customers, if you have set a rule to serve managed challenges to definitely automated bots, AI bots will also be challenged because custom rules run in a phase before Super Bot Fight Mode, which is the phase when the rule to block AI bots runs. -This behavior remains the same if the setting for verified, definitely automated, and likely bots is set to `block` or `allow`. If you have an action to `allow` for these rules, the request is not matched to any rule and proceeds to the next ruleset phase. Similarly, if the action is set to block, they will be blocked in the earlier phase and do not move on to match the AI rule at all. However, when the action is `challenge`, the request matches a rule and therefore will not be matched to any rules after. +This behavior remains the same if the setting for verified, definitely automated, and likely bots is set to `block` or `allow`. If you have an action to `allow` for these rules, the request is not matched to any rule and proceeds to the next ruleset phase. Similarly, if the action is set to `block`, they will be blocked in the earlier phase and do not move on to match the AI rule at all. However, when the action is `challenge`, the request matches a rule and therefore will not be matched to any rules after. For self-serve non-Bot Management customers, all rules for verified, definitely automated, and likely bots run in the phase following the AI bots rule. @@ -26,7 +26,7 @@ title: Rule phases flowchart LR accTitle: AI bots rule phases diagram accDescr: This diagram details the phases in which AI bots rules run. -A[Custom rules] --> B[Block AI bots rule] --> C[Allow verified bots rule] +A[Custom rules] --> B[Block AI bots
managed rule] --> C[Allow verified bots rule] ``` This feature is available on all Cloudflare plans.