Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,10 @@ AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk

## Compatibility with existing `robots.txt` files

Cloudflare will indepedently check whether your website has an existing `robots.txt` file. If your website already has one — verified by a HTTP `200` response — we will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
Cloudflare will independently check whether your website has an existing `robots.txt` file.

- If your website already has a `robots.txt` file — verified by a HTTP `200` response, we will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
- If your website does not have a `robots.txt` file, it will receive our managed block directives.

To implement a `robots.txt` file on your domain based on your plan:

Expand All @@ -33,4 +36,5 @@ To implement a `robots.txt` file on your domain based on your plan:
</Tabs>

## Availability

Managed `robots.txt` for AI crawlers is available on all plans.
3 changes: 3 additions & 0 deletions src/content/release-notes/bots.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ productLink: "/bots/"
productArea: Application security
productAreaLink: /fundamentals/reference/changelog/security/
entries:
- publish_date: "2025-07-02"
title: Managed robots.txt will prepend existing files
description: Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response.
- publish_date: "2025-05-14"
title: Anomaly detection events now receive a bot score of 2
description: Events detected by the [anomaly detection engine](/bots/concepts/bot-detection-engines/#anomaly-detection-enterprise) are now given a bot score of 2.
Expand Down
Loading