diff --git a/src/content/docs/bots/additional-configurations/managed-robots-txt.mdx b/src/content/docs/bots/additional-configurations/managed-robots-txt.mdx new file mode 100644 index 00000000000000..e5cb11f61d151e --- /dev/null +++ b/src/content/docs/bots/additional-configurations/managed-robots-txt.mdx @@ -0,0 +1,36 @@ +--- +pcx_content_type: reference +title: Direct AI crawlers with managed robots.txt +sidebar: + order: 8 + label: Managed robots.txt +--- + +import { Render, Tabs, TabItem } from "~/components" + +Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training. + +Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity. + +AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned. + +To implement a `robots.txt` file on your domain based on your plan: + + + + + + + + + + + + + +:::note +When you enable the `robots.txt` file on your domain, existing `robots.txt` files will be replaced. +::: + +## Availability +Managed `robots.txt` for AI crawlers is available on all plans. diff --git a/src/content/docs/bots/additional-configurations/static-resources.mdx b/src/content/docs/bots/additional-configurations/static-resources.mdx index a797334d0462ba..4e22248f610527 100644 --- a/src/content/docs/bots/additional-configurations/static-resources.mdx +++ b/src/content/docs/bots/additional-configurations/static-resources.mdx @@ -2,7 +2,7 @@ pcx_content_type: reference title: Static resource protection sidebar: - order: 8 + order: 9 --- diff --git a/src/content/docs/security/settings.mdx b/src/content/docs/security/settings.mdx index c4d1f2441df730..6dbb2a7b9539e7 100644 --- a/src/content/docs/security/settings.mdx +++ b/src/content/docs/security/settings.mdx @@ -100,15 +100,16 @@ This section allows you to configure multiple security-related settings. The fol | [Custom username and password location](/waf/detections/leaked-credentials/#custom-detection-locations) | **Security** > **Settings** | | [Custom content location](/waf/detections/malicious-uploads/#custom-scan-expressions) | **Security** > **Settings** | | [Custom sensitive data deployment](/waf/managed-rules/reference/sensitive-data-detection/#configure-in-the-dashboard) | **Security** > **Sensitive Data** | -| [Block definitely automated traffic](/bots/get-started/super-bot-fight-mode/#bot-settings) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | -| [Block likely bots](/bots/get-started/super-bot-fight-mode/#bot-settings) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | -| [Allow verified bots](/bots/get-started/super-bot-fight-mode/#bot-settings) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | -| [Static resource protection](/bots/additional-configurations/static-resources/) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | -| [Optimize for WordPress](/bots/troubleshooting/wordpress-loopback-issue/) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | -| [JavaScript detections](/bots/additional-configurations/javascript-detections/) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | +| [Block definitely automated traffic](/bots/get-started/super-bot-fight-mode/) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | +| [Block likely bots](/bots/get-started/super-bot-fight-mode/) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | +| [Managed `robots.txt`](/bots/additional-configurations/managed-robots-txt/) | **Security** > **Bots** > **Configure Bot Fight Mode
Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | +| [Allow verified bots](/bots/get-started/super-bot-fight-mode/) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | +| [Static resource protection](/bots/additional-configurations/static-resources/) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | +| [Optimize for WordPress](/bots/troubleshooting/wordpress-loopback-issue/) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | +| [JavaScript detections](/bots/additional-configurations/javascript-detections/) | **Security** > **Bots** > **Configure Super Bot Fight Mode
Security** > **Bots** > **Configure Bot Management** | | [Auto-update machine learning model](/bots/reference/machine-learning-models/) | **Security** > **Bots** > **Configure Bot Management** | | [Enable Security.txt](/security-center/infrastructure/security-file/) | **Security** > **Settings** | -| [Challenge Passage](/fundamentals/security/cloudflare-challenges/challenge-passage/) | **Security** > **Settings** | +| [Challenge Passage](/fundamentals/security/cloudflare-challenges/challenge-passage/) | **Security** > **Settings** | | [Browser Integrity Check](/waf/tools/browser-integrity-check/) | **Security** > **Settings** | | [Replace insecure JavaScript libraries](/waf/tools/replace-insecure-js-libraries/) | **Security** > **Settings** | | [Security Level](/waf/tools/security-level/) | **Security** > **Settings** | diff --git a/src/content/partials/bots/enable-managed-robots-txt.mdx b/src/content/partials/bots/enable-managed-robots-txt.mdx new file mode 100644 index 00000000000000..71e8210a53749f --- /dev/null +++ b/src/content/partials/bots/enable-managed-robots-txt.mdx @@ -0,0 +1,11 @@ +--- +inputParameters: params1 + +--- + +import { Markdown } from "~/components" + +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain. +2. Go to **Security** > **Bots**. +3. Select **Configure {props.one}**. +4. On **Manage bot traffic with robots.txt**, select the toggle. \ No newline at end of file diff --git a/src/content/release-notes/bots.yaml b/src/content/release-notes/bots.yaml index e85e4362c0f2d3..de29fc226548a4 100644 --- a/src/content/release-notes/bots.yaml +++ b/src/content/release-notes/bots.yaml @@ -5,6 +5,10 @@ productLink: "/bots/" productArea: Application security productAreaLink: /fundamentals/reference/changelog/security/ entries: + - publish_date: "2025-04-28" + title: Managed robots.txt is now available + description: |- + Direct AI crawlers on what they can and cannot scrape from your website or application by [implementing a `robots.txt` file](/bots/additional-configurations/managed-robots-txt/) to your domain. - publish_date: "2025-04-24" title: Bot Detection Alerts are now available description: |-