Skip to content

Commit 440b4a2

Browse files
committed
small tweaks
1 parent a3ee209 commit 440b4a2

File tree

1 file changed

+3
-3
lines changed

1 file changed

+3
-3
lines changed

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,9 @@ import { Render, Tabs, TabItem } from "~/components";
1010

1111
Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.
1212

13-
Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.
13+
Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes.
1414

15-
AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned.
15+
AI bots are expected to follow the `robots.txt` directives.
1616

1717
## Compatibility with existing `robots.txt` files
1818

@@ -22,7 +22,7 @@ Cloudflare will independently check whether your website has an existing `robots
2222

2323
If your website already has a `robots.txt` file — verified by a HTTP `200` response — Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
2424

25-
Without this feature enabled, the `robots.txt` content of `crawlstop.com` would be:
25+
For example, without this feature enabled, the `robots.txt` content of `crawlstop.com` would be:
2626

2727
```txt
2828
# This robots.txt is managed using Cloudflare's managed robots.txt feature

0 commit comments

Comments
 (0)