Skip to content

[Bots] CSP updates #25372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Sep 24, 2025
Merged

[Bots] CSP updates #25372

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
90f4a4e
remove dash
patriciasantaana Sep 23, 2025
53496e0
typos
patriciasantaana Sep 23, 2025
0348518
correct dashbutton
patriciasantaana Sep 23, 2025
59dc539
remove dashes
patriciasantaana Sep 23, 2025
07cb611
update naming
patriciasantaana Sep 24, 2025
febac95
fix
patriciasantaana Sep 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions src/content/docs/bots/additional-configurations/managed-robots-txt.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
pcx_content_type: reference
title: Instruct AI crawlers with managed robots.txt
title: robots.txt setting
sidebar:
order: 10
label: Managed robots.txt
label: robots.txt setting
---

import { Render, Tabs, TabItem, Steps, DashButton } from "~/components";
Expand Down Expand Up @@ -41,7 +41,7 @@ With the managed `robots.txt` enabled, Cloudflare will prepend our managed conte

```txt title="Feature enabled"
# As a condition of accessing this website, you agree to abide by the
# following content-signals:
# following content signals:

# (a) If a content-signal = yes, you may collect content for the
# corresponding use.
Expand All @@ -61,14 +61,14 @@ With the managed `robots.txt` enabled, Cloudflare will prepend our managed conte
# content for generative AI search answers).
# ai-train: training or fine-tuning AI models.

# ANY RESTRICTIONS EXPRESSED VIA CONTENT-SIGNALS ARE EXPRESS RESERVATIONS OF
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.

# BEGIN Cloudflare Managed content

User-Agent: *
Content-signal: search=yes,ai-train=no
Content-signal: search=yes, ai-train=no
Allow: /

User-agent: Amazonbot
Expand Down Expand Up @@ -127,8 +127,8 @@ To implement a `robots.txt` file on your domain:

<DashButton url="/?to=/:account/:zone/security/settings" />
2. Filter by **Bot traffic**.
3. Go to **Instruct AI bot traffic with robots.txt**.
4. Turn **Instruct AI bot traffic with robots.txt** on.
3. Go to **robots.txt setting**.
4. Turn **robots.txt setting** on.
</Steps>
</TabItem>
</Tabs>
Expand All @@ -141,7 +141,7 @@ This file only outlines the Content Signals framework. It does not express your

```txt title="Content Signals Policy"
# As a condition of accessing this website, you agree to abide by the
# following content-signals:
# following content signals:

# (a) If a content-signal = yes, you may collect content for the
# corresponding use.
Expand All @@ -161,16 +161,16 @@ This file only outlines the Content Signals framework. It does not express your
# content for generative AI search answers).
# ai-train: training or fine-tuning AI models.

# ANY RESTRICTIONS EXPRESSED VIA CONTENT-SIGNALS ARE EXPRESS RESERVATIONS OF
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
```

Cloudflare's Content Signals Policy is included by default in the `robots.txt` file when you turn on **Instruct AI bot traffic with robots.txt**.
Cloudflare's Content Signals Policy is included by default in the `robots.txt` file when you turn on **robots.txt setting**.

If you would like to opt out of displaying the policy in your `robots.txt` file, you can uncheck **Display Content Signals Policy** under **Control AI Crawlers** in your zone's overview.

<DashButton url="/?to=/:account/:zone/security/overview" />
<DashButton url="/?to=/:account/:zone/" />

Alternatively, you can use [Security Settings](#implementation).

Expand Down
Loading