managed robots.txt

Author: patriciasantaana

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

@@ -0,0 +1,36 @@
+---
+pcx_content_type: reference
+title: Direct AI crawlers with managed robots.txt
+sidebar:
+  order: 8
+  label: Managed robots.txt
+---
+
+import { Render, Tabs, TabItem } from "~/components"
+
+Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training. 
+
+Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity. 
+
+AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned. 
+
+To implement a `robots.txt` file on your domain based on your plan:
+
+<Tabs>
+  <TabItem label="Bot Fight Mode">
+    <Render file="enable-managed-robots-txt" params={{ one: "Bot Fight Mode" }} />
+  </TabItem>
+  <TabItem label="Super Bot Fight Mode">
+    <Render file="enable-managed-robots-txt" params={{ one: "Super Bot Fight Mode" }} />
+  </TabItem>
+    <TabItem label="Bot Management for Enterprise">
+    <Render file="enable-managed-robots-txt" params={{ one: "Bot Management" }} />
+  </TabItem>
+</Tabs>
+
+:::note
+When you select the toggle to implement a `robots.txt` file on your domain, existing `robots.txt` files will be replaced.
+:::
+
+## Availability
+Managed `robots.txt` for AI crawlers is available on all plans.

src/content/docs/bots/additional-configurations/static-resources.mdx

@@ -2,7 +2,7 @@
 pcx_content_type: reference
 title: Static resource protection
 sidebar:
-  order: 8
+  order: 9
 
 ---
 

src/content/partials/bots/enable-managed-robots-txt.mdx

@@ -0,0 +1,11 @@
+---
+inputParameters: params1
+
+---
+
+import { Markdown } from "~/components"
+
+1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
+2. Go to **Security** > **Bots**.
+3. Select **Configure {props.one}**.
+4. On **Manage bot traffic with robots.txt**, select the toggle.