Skip to content

Commit ae6fab0

Browse files
committed
[Bots] Add robots.txt example file in iframe
1 parent cfc2fa6 commit ae6fab0

File tree

1 file changed

+12
-2
lines changed

1 file changed

+12
-2
lines changed

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,6 @@ import { Render, Tabs, TabItem, Steps } from "~/components";
1010

1111
Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.
1212

13-
Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes.
14-
1513
AI bots are expected to follow the `robots.txt` directives.
1614

1715
:::note
@@ -39,6 +37,18 @@ Sitemap: https://www.crawlstop.com/sitemap.xml
3937

4038
With the managed `robots.txt` enabled, Cloudflare will prepend our managed content before your original content, resulting in what you can view at https://crawlstop.com/robots.txt.
4139

40+
**Robots.txt example**
41+
<div style="position: relative; padding-top: 56.25%; border: 1px solid orange; border-radius: 5px">
42+
<iframe
43+
src="https://crawlstop.com/robots.txt"
44+
style="border: none; position: absolute; top: 0; left: 0; height: 100%; width: 100%;"
45+
allow="accelerometer; gyroscope; autoplay; encrypted-media; picture-in-picture;"
46+
allowfullscreen="true"
47+
title="crawltop.com robots.txt file"
48+
>
49+
</iframe>
50+
</div>
51+
4252
### No robots.txt file
4353

4454
If your website does not have a `robots.txt` file, Cloudflare creates a new file with our managed block directives and serves it for you.

0 commit comments

Comments
 (0)