Skip to content

[Bots] block AI bots updates #17826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 29, 2024
Merged

[Bots] block AI bots updates #17826

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
08dd85f
block ai bots updates
patriciasantaana Oct 28, 2024
123aeb1
space
patriciasantaana Oct 28, 2024
7001e2a
update placement and link
patriciasantaana Oct 28, 2024
8a88742
wording edit
patriciasantaana Oct 29, 2024
a70da2b
Apply suggestions from code review
patriciasantaana Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions src/content/docs/bots/get-started/biz-and-ent.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,6 @@ If you have an Enterprise subscription without Bot Management, your application

## Block AI bots

<Render file="ai-bots-definition" />

<Render file="block-ai-bots-enable" params={{ one: "Super Bot Fight Mode" }} />

:::note
Expand Down
2 changes: 0 additions & 2 deletions src/content/docs/bots/get-started/bm-subscription.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,6 @@ If you are not seeing Bot Management enabled on your zone or if you still see **

## Block AI bots

<Render file="ai-bots-definition" />

<Render file="block-ai-bots-enable" params={{ one: "Bot Management" }} />

:::note
Expand Down
2 changes: 0 additions & 2 deletions src/content/docs/bots/get-started/free.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,6 @@ To disable Bot Fight Mode:

## Block AI bots

<Render file="ai-bots-definition" />

<Render file="block-ai-bots-enable" params={{ one: "Bot Fight Mode" }} />

:::note
Expand Down
2 changes: 0 additions & 2 deletions src/content/docs/bots/get-started/pro.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,6 @@ Super Bot Fight Mode is included in your Pro subscription. When enabled, the pro

## Block AI bots

<Render file="ai-bots-definition" />

<Render file="block-ai-bots-enable" params={{ one: "Super Bot Fight Mode" }} />

:::note
Expand Down
26 changes: 25 additions & 1 deletion src/content/partials/bots/ai-bots-definition.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,30 @@

---

You can block artificial intelligence (AI) bots, crawlers, and scrapers from scraping your website content and training large language models (LLM) to recreate it without your permission. When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that respect `robots.txt` and crawl rate, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules.
You can block artificial intelligence (AI) bots, crawlers, and scrapers from scraping your website content and training large language models (LLM) to recreate it without your permission.

:::note
Some AI bots overlap with definitely automated bots and verified bots, the latter becoming verified AI bots.
:::

When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with `robots.txt` and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules.

The rule to block AI bots takes precedence over all other Super Bot Fight Mode rules. For example, if you have enabled **Block AI bots** and **Allow verified bots**, verified AI bots will also be blocked even if you allow other verified bots on your website or application.

For Bot Management customers, if you have set a rule to serve managed challenges to definitely automated bots, AI bots will also be challenged because custom rules run in a phase before Super Bot Fight Mode, which is the phase when the rule to block AI bots runs.

This behavior remains the same if the setting for verified, definitely automated, and likely bots is set to `block` or `allow`. If you have an action to `allow` for these rules, the request is not matched to any rule and proceeds to the next ruleset phase. Similarly, if the action is set to block, they will be blocked in the earlier phase and do not move on to match the AI rule at all. However, when the action is `challenge`, the request matches a rule and therefore will not be matched to any rules after.

For self-serve non-Bot Management customers, all rules for verified, definitely automated, and likely bots run in the phase following the AI bots rule.

```mermaid
---
title: Rule phases
---
flowchart LR
accTitle: AI bots rule phases diagram
accDescr: This diagram details the phases in which AI bots rules run.
A[Custom rules] --> B[Block AI bots rule] --> C[Allow verified bots rule]
```

This feature is available on all Cloudflare plans.
4 changes: 3 additions & 1 deletion src/content/partials/bots/block-ai-bots-enable.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ inputParameters: param1

import { Markdown } from "~/components"

To block [AI bots](/bots/concepts/bot/#ai-bots):

1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
2. Go to **Security** > **Bots**.
3. Select **Configure {props.one}**.
4. Enable **Block AI bots**.
4. Enable **Block AI bots**.
Loading