You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.
11
+
Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.
12
12
13
-
Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.
13
+
Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.
14
14
15
-
AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned.
15
+
AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned.
16
16
17
17
## Compatibility with existing `robots.txt` files
18
18
19
-
Cloudflare will independently check whether your website has an existing `robots.txt` file.
19
+
Cloudflare will independently check whether your website has an existing `robots.txt` file and update the behavior of this feature based on your website.
20
20
21
-
- If your website already has a `robots.txt` file — verified by a HTTP `200` response — Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
22
-
- If your website does not have a `robots.txt` file, it will receive Cloudflare's managed block directives.
21
+
### Existing robots.txt file
22
+
23
+
If your website already has a `robots.txt` file — verified by a HTTP `200` response — Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
24
+
25
+
Without this feature enabled, the `robots.txt` content of `crawlstop.com` would be:
26
+
27
+
```txt
28
+
# This robots.txt is managed using Cloudflare's managed robots.txt feature
# Cloudflare will prepend their managed robots.txt directives before this content
32
+
```
33
+
34
+
With the managed `robots.txt` enabled, Cloudflare will prepend our managed content before your original content, resulting in what you can view at https://crawlstop.com/robots.txt.
35
+
36
+
### No robots.txt file
37
+
38
+
If your website does not have a `robots.txt` file, Cloudflare creates a new file with our managed block directives and serves it for you.
39
+
40
+
## Implementation
23
41
24
42
To implement a `robots.txt` file on your domain based on your plan:
0 commit comments