Skip to content

Commit 94d416c

Browse files
Update managed-robots-txt.mdx (#23205)
* Update managed-robots-txt.mdx * Apply suggestions from code review --------- Co-authored-by: Patricia Santa Ana <[email protected]>
1 parent 08f82c6 commit 94d416c

File tree

1 file changed

+4
-4
lines changed

1 file changed

+4
-4
lines changed

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scr
1414

1515
AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned.
1616

17+
## Compatibility with existing `robots.txt` files
18+
19+
Cloudflare will indepedently check whether your website has an existing `robots.txt` file. If your website already has one — verified by a HTTP `200` response — we will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
20+
1721
To implement a `robots.txt` file on your domain based on your plan:
1822

1923
<Tabs>
@@ -28,9 +32,5 @@ To implement a `robots.txt` file on your domain based on your plan:
2832
</TabItem>
2933
</Tabs>
3034

31-
:::note
32-
When you enable the `robots.txt` file on your domain, existing `robots.txt` files will be replaced.
33-
:::
34-
3535
## Availability
3636
Managed `robots.txt` for AI crawlers is available on all plans.

0 commit comments

Comments
 (0)