Skip to content

Commit eaf35e8

Browse files
patriciasantaanamarciocloudflare
authored and
daisyfaithauma
committed
[Bots] Managed robots.txt (#22026)
* managed robots.txt * security settings * release notes entry * Update src/content/docs/bots/additional-configurations/managed-robots-txt.mdx Co-authored-by: marciocloudflare <[email protected]> * Apply suggestions from code review --------- Co-authored-by: marciocloudflare <[email protected]>
1 parent 70fa3d3 commit eaf35e8

File tree

5 files changed

+60
-8
lines changed

5 files changed

+60
-8
lines changed

‎src/content/docs/bots/additional-configurations/managed-robots-txt.mdx‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
pcx_content_type: reference
3+
title: Direct AI crawlers with managed robots.txt
4+
sidebar:
5+
order: 8
6+
label: Managed robots.txt
7+
---
8+
9+
import { Render, Tabs, TabItem } from "~/components"
10+
11+
Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.
12+
13+
Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.
14+
15+
AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned.
16+
17+
To implement a `robots.txt` file on your domain based on your plan:
18+
19+
<Tabs>
20+
<TabItem label="Bot Fight Mode">
21+
<Render file="enable-managed-robots-txt" params={{ one: "Bot Fight Mode" }} />
22+
</TabItem>
23+
<TabItem label="Super Bot Fight Mode">
24+
<Render file="enable-managed-robots-txt" params={{ one: "Super Bot Fight Mode" }} />
25+
</TabItem>
26+
<TabItem label="Bot Management for Enterprise">
27+
<Render file="enable-managed-robots-txt" params={{ one: "Bot Management" }} />
28+
</TabItem>
29+
</Tabs>
30+
31+
:::note
32+
When you enable the `robots.txt` file on your domain, existing `robots.txt` files will be replaced.
33+
:::
34+
35+
## Availability
36+
Managed `robots.txt` for AI crawlers is available on all plans.

‎src/content/docs/bots/additional-configurations/static-resources.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
pcx_content_type: reference
33
title: Static resource protection
44
sidebar:
5-
order: 8
5+
order: 9
66

77
---
88

‎src/content/docs/security/settings.mdx‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -100,15 +100,16 @@ This section allows you to configure multiple security-related settings. The fol
100100
| [Custom username and password location](/waf/detections/leaked-credentials/#custom-detection-locations) | **Security** > **Settings** |
101101
| [Custom content location](/waf/detections/malicious-uploads/#custom-scan-expressions) | **Security** > **Settings** |
102102
| [Custom sensitive data deployment](/waf/managed-rules/reference/sensitive-data-detection/#configure-in-the-dashboard) | **Security** > **Sensitive Data** |
103-
| [Block definitely automated traffic](/bots/get-started/super-bot-fight-mode/#bot-settings) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
104-
| [Block likely bots](/bots/get-started/super-bot-fight-mode/#bot-settings) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
105-
| [Allow verified bots](/bots/get-started/super-bot-fight-mode/#bot-settings) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
106-
| [Static resource protection](/bots/additional-configurations/static-resources/) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
107-
| [Optimize for WordPress](/bots/troubleshooting/wordpress-loopback-issue/) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
108-
| [JavaScript detections](/bots/additional-configurations/javascript-detections/) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
103+
| [Block definitely automated traffic](/bots/get-started/super-bot-fight-mode/) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
104+
| [Block likely bots](/bots/get-started/super-bot-fight-mode/) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
105+
| [Managed `robots.txt`](/bots/additional-configurations/managed-robots-txt/) | **Security** > **Bots** > **Configure Bot Fight Mode<br/>Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
106+
| [Allow verified bots](/bots/get-started/super-bot-fight-mode/) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
107+
| [Static resource protection](/bots/additional-configurations/static-resources/) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
108+
| [Optimize for WordPress](/bots/troubleshooting/wordpress-loopback-issue/) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
109+
| [JavaScript detections](/bots/additional-configurations/javascript-detections/) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
109110
| [Auto-update machine learning model](/bots/reference/machine-learning-models/) | **Security** > **Bots** > **Configure Bot Management** |
110111
| [Enable Security.txt](/security-center/infrastructure/security-file/) | **Security** > **Settings** |
111-
| [Challenge Passage](/fundamentals/security/cloudflare-challenges/challenge-passage/) | **Security** > **Settings** |
112+
| [Challenge Passage](/fundamentals/security/cloudflare-challenges/challenge-passage/) | **Security** > **Settings** |
112113
| [Browser Integrity Check](/waf/tools/browser-integrity-check/) | **Security** > **Settings** |
113114
| [Replace insecure JavaScript libraries](/waf/tools/replace-insecure-js-libraries/) | **Security** > **Settings** |
114115
| [Security Level](/waf/tools/security-level/) | **Security** > **Settings** |

‎src/content/partials/bots/enable-managed-robots-txt.mdx‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
inputParameters: params1
3+
4+
---
5+
6+
import { Markdown } from "~/components"
7+
8+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
9+
2. Go to **Security** > **Bots**.
10+
3. Select **Configure {props.one}**.
11+
4. On **Manage bot traffic with robots.txt**, select the toggle.

‎src/content/release-notes/bots.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@ productLink: "/bots/"
55
productArea: Application security
66
productAreaLink: /fundamentals/reference/changelog/security/
77
entries:
8+
- publish_date: "2025-04-28"
9+
title: Managed robots.txt is now available
10+
description: |-
11+
Direct AI crawlers on what they can and cannot scrape from your website or application by [implementing a `robots.txt` file](/bots/additional-configurations/managed-robots-txt/) to your domain.
812
- publish_date: "2025-04-24"
913
title: Bot Detection Alerts are now available
1014
description: |-

0 commit comments

Comments
 (0)