Skip to content

Commit 2f54075

Browse files
text edits
1 parent d5a9250 commit 2f54075

File tree

1 file changed

+5
-1
lines changed

1 file changed

+5
-1
lines changed

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,10 @@ AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk
1616

1717
## Compatibility with existing `robots.txt` files
1818

19-
Cloudflare will indepedently check whether your website has an existing `robots.txt` file. If your website already has one — verified by a HTTP `200` response — we will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
19+
Cloudflare will independently check whether your website has an existing `robots.txt` file.
20+
21+
- If your website already has a `robots.txt` file — verified by a HTTP `200` response, we will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
22+
- If your website does not have a `robots.txt` file, it will receive our managed block directives.
2023

2124
To implement a `robots.txt` file on your domain based on your plan:
2225

@@ -33,4 +36,5 @@ To implement a `robots.txt` file on your domain based on your plan:
3336
</Tabs>
3437

3538
## Availability
39+
3640
Managed `robots.txt` for AI crawlers is available on all plans.

0 commit comments

Comments
 (0)