@@ -26,7 +26,7 @@ If your website already has a `robots.txt` file — verified by a HTTP `200` res
2626
2727For example, without this feature enabled, the ` robots.txt ` content of ` crawlstop.com ` would be:
2828
29- ``` txt
29+ ``` txt title="Feature not enabled"
3030User-agent: *
3131Disallow: /lp
3232Disallow: /feedback
@@ -37,16 +37,53 @@ Sitemap: https://www.crawlstop.com/sitemap.xml
3737
3838With the managed ` robots.txt ` enabled, Cloudflare will prepend our managed content before your original content, resulting in what you can view at https://www.crawlstop.com/robots.txt .
3939
40- ** Robots.txt example**
41- <div style = " position: relative; padding-top: 56.25%; border: 1px solid orange; border-radius: 5px" >
42- <iframe
43- src = " https://www.crawlstop.com/robots.txt"
44- style = " border: none; position: absolute; top: 0; left: 0; height: 100%; width: 100%;"
45- allowfullscreen = " true"
46- title = " crawltop.com robots.txt file"
47- >
48- </iframe >
49- </div >
40+ ``` txt title="Feature enabled"
41+ # NOTICE: The collection of content and other data on this
42+ # site through automated means, including any device, tool,
43+ # or process designed to data mine or scrape content, is
44+ # prohibited except (1) for the purpose of search engine indexing or
45+ # artificial intelligence retrieval augmented generation or (2) with express
46+ # written permission from this site’s operator.
47+
48+ # To request permission to license our intellectual
49+ # property and/or other materials, please contact this
50+ # site’s operator directly.
51+
52+ # BEGIN Cloudflare Managed content
53+
54+ User-agent: Amazonbot
55+ Disallow: /
56+
57+ User-agent: Applebot-Extended
58+ Disallow: /
59+
60+ User-agent: Bytespider
61+ Disallow: /
62+
63+ User-agent: CCBot
64+ Disallow: /
65+
66+ User-agent: ClaudeBot
67+ Disallow: /
68+
69+ User-agent: Google-Extended
70+ Disallow: /
71+
72+ User-agent: GPTBot
73+ Disallow: /
74+
75+ User-agent: meta-externalagent
76+ Disallow: /
77+
78+ # END Cloudflare Managed Content
79+ User-agent: *
80+ Disallow: /lp
81+ Disallow: /feedback
82+ Disallow: /langtest
83+
84+
85+ Sitemap: https://www.crawlstop.com/sitemap.xml
86+ ```
5087
5188### No robots.txt file
5289
0 commit comments