Skip to content

Commit a3ee209

Browse files
committed
[Bots] Update managed robots.txt
1 parent 5ed518c commit a3ee209

File tree

1 file changed

+43
-16
lines changed

1 file changed

+43
-16
lines changed

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Lines changed: 43 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -6,33 +6,60 @@ sidebar:
66
label: Managed robots.txt
77
---
88

9-
import { Render, Tabs, TabItem } from "~/components"
9+
import { Render, Tabs, TabItem } from "~/components";
1010

11-
Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.
11+
Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.
1212

13-
Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.
13+
Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.
1414

15-
AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned.
15+
AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned.
1616

1717
## Compatibility with existing `robots.txt` files
1818

19-
Cloudflare will independently check whether your website has an existing `robots.txt` file.
19+
Cloudflare will independently check whether your website has an existing `robots.txt` file and update the behavior of this feature based on your website.
2020

21-
- If your website already has a `robots.txt` file — verified by a HTTP `200` response — Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
22-
- If your website does not have a `robots.txt` file, it will receive Cloudflare's managed block directives.
21+
### Existing robots.txt file
22+
23+
If your website already has a `robots.txt` file — verified by a HTTP `200` response — Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
24+
25+
Without this feature enabled, the `robots.txt` content of `crawlstop.com` would be:
26+
27+
```txt
28+
# This robots.txt is managed using Cloudflare's managed robots.txt feature
29+
# See: https://developers.cloudflare.com/bots/additional-configurations/managed-robots.txt/
30+
31+
# Cloudflare will prepend their managed robots.txt directives before this content
32+
```
33+
34+
With the managed `robots.txt` enabled, Cloudflare will prepend our managed content before your original content, resulting in what you can view at https://crawlstop.com/robots.txt.
35+
36+
### No robots.txt file
37+
38+
If your website does not have a `robots.txt` file, Cloudflare creates a new file with our managed block directives and serves it for you.
39+
40+
## Implementation
2341

2442
To implement a `robots.txt` file on your domain based on your plan:
2543

2644
<Tabs>
27-
<TabItem label="Bot Fight Mode">
28-
<Render file="enable-managed-robots-txt" params={{ one: "Bot Fight Mode" }} />
29-
</TabItem>
30-
<TabItem label="Super Bot Fight Mode">
31-
<Render file="enable-managed-robots-txt" params={{ one: "Super Bot Fight Mode" }} />
32-
</TabItem>
33-
<TabItem label="Bot Management for Enterprise">
34-
<Render file="enable-managed-robots-txt" params={{ one: "Bot Management" }} />
35-
</TabItem>
45+
<TabItem label="Bot Fight Mode">
46+
<Render
47+
file="enable-managed-robots-txt"
48+
params={{ one: "Bot Fight Mode" }}
49+
/>
50+
</TabItem>
51+
<TabItem label="Super Bot Fight Mode">
52+
<Render
53+
file="enable-managed-robots-txt"
54+
params={{ one: "Super Bot Fight Mode" }}
55+
/>
56+
</TabItem>
57+
<TabItem label="Bot Management for Enterprise">
58+
<Render
59+
file="enable-managed-robots-txt"
60+
params={{ one: "Bot Management" }}
61+
/>
62+
</TabItem>
3663
</Tabs>
3764

3865
## Availability

0 commit comments

Comments
 (0)