Skip to content

Commit c58b49e

Browse files
authored
Update managed-robots-txt.mdx (#25306)
remove iframe b/c of engineering limitations
1 parent e5b116b commit c58b49e

File tree

1 file changed

+48
-11
lines changed

1 file changed

+48
-11
lines changed

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Lines changed: 48 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ If your website already has a `robots.txt` file — verified by a HTTP `200` res
2626

2727
For example, without this feature enabled, the `robots.txt` content of `crawlstop.com` would be:
2828

29-
```txt
29+
```txt title="Feature not enabled"
3030
User-agent: *
3131
Disallow: /lp
3232
Disallow: /feedback
@@ -37,16 +37,53 @@ Sitemap: https://www.crawlstop.com/sitemap.xml
3737

3838
With the managed `robots.txt` enabled, Cloudflare will prepend our managed content before your original content, resulting in what you can view at https://www.crawlstop.com/robots.txt.
3939

40-
**Robots.txt example**
41-
<div style="position: relative; padding-top: 56.25%; border: 1px solid orange; border-radius: 5px">
42-
<iframe
43-
src="https://www.crawlstop.com/robots.txt"
44-
style="border: none; position: absolute; top: 0; left: 0; height: 100%; width: 100%;"
45-
allowfullscreen="true"
46-
title="crawltop.com robots.txt file"
47-
>
48-
</iframe>
49-
</div>
40+
```txt title="Feature enabled"
41+
# NOTICE: The collection of content and other data on this
42+
# site through automated means, including any device, tool,
43+
# or process designed to data mine or scrape content, is
44+
# prohibited except (1) for the purpose of search engine indexing or
45+
# artificial intelligence retrieval augmented generation or (2) with express
46+
# written permission from this site’s operator.
47+
48+
# To request permission to license our intellectual
49+
# property and/or other materials, please contact this
50+
# site’s operator directly.
51+
52+
# BEGIN Cloudflare Managed content
53+
54+
User-agent: Amazonbot
55+
Disallow: /
56+
57+
User-agent: Applebot-Extended
58+
Disallow: /
59+
60+
User-agent: Bytespider
61+
Disallow: /
62+
63+
User-agent: CCBot
64+
Disallow: /
65+
66+
User-agent: ClaudeBot
67+
Disallow: /
68+
69+
User-agent: Google-Extended
70+
Disallow: /
71+
72+
User-agent: GPTBot
73+
Disallow: /
74+
75+
User-agent: meta-externalagent
76+
Disallow: /
77+
78+
# END Cloudflare Managed Content
79+
User-agent: *
80+
Disallow: /lp
81+
Disallow: /feedback
82+
Disallow: /langtest
83+
84+
85+
Sitemap: https://www.crawlstop.com/sitemap.xml
86+
```
5087

5188
### No robots.txt file
5289

0 commit comments

Comments
 (0)