Skip to content

Commit f211c09

Browse files
committed
Separating "Block AI bots" as a new page
1 parent 27823e4 commit f211c09

File tree

6 files changed

+36
-13
lines changed

6 files changed

+36
-13
lines changed
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
pcx_content_type: reference
3+
title: Block AI Bots
4+
sidebar:
5+
order: 8
6+
label: Block AI Bots
7+
---
8+
9+
import { Render, Tabs, TabItem } from "~/components"
10+
11+
You can choose to block AI bots by activating **Block AI bots**. Activating this setting will block Definitely Automated bots and Verified AI Bots, such as AI Search, AI Assistant, AI Crawler, or an AI Archiver.
12+
13+
To block [AI bots](/bots/concepts/bot/#ai-bots):
14+
15+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
16+
2. Go to **Security** > **Settings**.
17+
3. Filter by **Bot traffic**.
18+
4. Go to **Block AI bots**.
19+
5. Under **Configurations**, select the edit icon. Choose from:
20+
- **Block only on pages with ads**: Use this option if you wish to only block AI bots on pages you monetize. Cloudflare automatically detects whether ads are present on a page, and only block on hostnames that contain those ad units.
21+
- **Block on all pages**: Use this option if you wish to block AI bots on all your pages.
22+
- **Do not block (off)**: Use this option if you wish to allow AI bots on all your pages.
23+
6. Select **Save** to save your configuration.

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,23 +2,23 @@
22
pcx_content_type: reference
33
title: Direct AI crawlers with managed robots.txt
44
sidebar:
5-
order: 8
5+
order: 10
66
label: Managed robots.txt
77
---
88

99
import { Render, Tabs, TabItem } from "~/components"
1010

11-
Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.
11+
Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.
1212

13-
Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.
13+
Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.
1414

15-
AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned.
15+
AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned.
1616

1717
## Compatibility with existing `robots.txt` files
1818

19-
Cloudflare will independently check whether your website has an existing `robots.txt` file.
19+
Cloudflare will independently check whether your website has an existing `robots.txt` file.
2020

21-
- If your website already has a `robots.txt` file — verified by a HTTP `200` response — Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
21+
- If your website already has a `robots.txt` file — verified by a HTTP `200` response — Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
2222
- If your website does not have a `robots.txt` file, it will receive Cloudflare's managed block directives.
2323

2424
To implement a `robots.txt` file on your domain based on your plan:

src/content/docs/bots/additional-configurations/static-resources.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
pcx_content_type: reference
33
title: Static resource protection
44
sidebar:
5-
order: 9
5+
order: 12
66

77
---
88

@@ -12,7 +12,7 @@ Pro, Business, and Enterprise customers can use Cloudflare's bot solutions to pr
1212

1313
:::caution[Warning]
1414

15-
If you enable static resource protection, you may block good bots — like mail clients — that routinely fetch static resources. Make sure you understand your existing infrastructure before enabling this feature.
15+
If you enable static resource protection, you may block good bots — like mail clients — that routinely fetch static resources. Make sure you understand your existing infrastructure before enabling this feature.
1616
:::
1717

1818
## Super Bot Fight Mode

src/content/docs/bots/get-started/bot-fight-mode.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,11 @@ To disable Bot Fight Mode:
3535

3636
## Block AI bots
3737

38-
<Render file="block-ai-bots-enable" params={{ one: "Bot Fight Mode" }} />
38+
Refer to [Block AI bots](/bots/additional-configurations/block-ai-bots/).
3939

4040
:::note
4141

42-
You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/).
42+
You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/).
4343
:::
4444

4545
## Visibility

src/content/docs/bots/get-started/bot-management.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ Cloudflare recommends that you deploy the following basic settings and customize
4444

4545
### Block AI Bots
4646

47-
<Render file="block-ai-bots-enable" params={{ one: "Bot Management" }} />
47+
Refer to [Block AI bots](/bots/additional-configurations/block-ai-bots/).
4848

4949
:::note
5050
You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/).

src/content/docs/bots/get-started/super-bot-fight-mode.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,11 +42,11 @@ Accounts with an Enterprise subscription but not the [Bot Management add-on](/bo
4242

4343
## Block AI bots
4444

45-
<Render file="block-ai-bots-enable" params={{ one: "Super Bot Fight Mode" }} />
45+
Refer to [Block AI bots](/bots/additional-configurations/block-ai-bots/).
4646

4747
:::note
4848

49-
You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/).
49+
You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/).
5050
:::
5151

5252
## Analytics

0 commit comments

Comments
 (0)