Skip to content

Commit 6169e92

Browse files
committed
Renaming AI Audit to AI Crawl Control
1 parent fa0dc9d commit 6169e92

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+162
-161
lines changed

.github/CODEOWNERS

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@
100100
/src/content/docs/d1/ @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @harshil1712 @cloudflare/pcx-technical-writing
101101
/src/content/release-notes/d1.yaml @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @cloudflare/pcx-technical-writing
102102
/src/content/partials/d1/ @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @harshil1712 @cloudflare/pcx-technical-writing
103-
/src/content/docs/ai-audit/ @oxyjun @kodster28 @cloudflare/pcx-technical-writing
103+
/src/content/docs/ai-crawl-control/ @oxyjun @kodster28 @cloudflare/pcx-technical-writing
104104
/src/content/docs/durable-objects/ @elithrar @vy-ton @joshthoward @oxyjun @harshil1712 @mikenomitch @cloudflare/pcx-technical-writing @cloudflare/workers-runtime-1
105105
/src/content/release-notes/durable-objects.yaml @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @cloudflare/pcx-technical-writing
106106
/src/content/docs/email-routing/ @cloudflare/pcx-technical-writing

public/__redirects

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -149,9 +149,10 @@
149149
/ai/ /use-cases/ai/ 301
150150

151151
# ai-audit
152-
/ai-audit/features/detect-ai-crawlers/ /ai-audit/features/analyze-ai-crawlers/ 301
153-
/ai-audit/features/enforce-robots-txt/ /ai-audit/features/analyze-ai-crawlers/ 301
154-
/ai-audit/features/analyze-ai-crawlers/ /ai-audit/features/analyze-ai-traffic/ 301
152+
/ai-audit/features/detect-ai-crawlers/ /ai-crawl-control/features/analyze-ai-crawlers/ 301
153+
/ai-audit/features/enforce-robots-txt/ /ai-crawl-control/features/analyze-ai-crawlers/ 301
154+
/ai-audit/features/analyze-ai-crawlers/ /ai-crawl-control/features/analyze-ai-traffic/ 301
155+
/ai-audit/* /ai-crawl-control/:splat 301
155156

156157
# AI Gateway
157158
/ai-gateway/get-started/configuring-settings/ /ai-gateway/get-started/ 301

src/content/changelog/ai-audit/2025-07-01-pay-per-crawl.mdx

Lines changed: 0 additions & 22 deletions
This file was deleted.

src/content/changelog/ai-audit/2025-07-01-refresh.mdx

Lines changed: 0 additions & 19 deletions
This file was deleted.
Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
11
---
2-
title: AI Audit
2+
title: AI Crawl Control
33
description: AI Audit is available to all customers
44
date: 2024-09-23
55
---
66

7-
Every site on Cloudflare now has access to [**AI Audit**](/ai-audit/), which summarizes the crawling behavior of popular and known AI services.
7+
Every site on Cloudflare now has access to [**AI Audit**](/ai-crawl-control/), which summarizes the crawling behavior of popular and known AI services.
88

99
You can use this data to:
1010

1111
- Understand how and how often crawlers access your site (and which content is the most popular).
1212
- Block specific AI bots accessing your site.
1313
- Use Cloudflare to enforce your `robots.txt` policy via an automatic WAF rule.
1414

15-
![View AI bot activity with AI Audit](~/assets/images/changelog/ai-audit/ai-audit-overview.png)
15+
![View AI bot activity with AI Audit](~/assets/images/changelog/ai-crawl-control/ai-crawl-control-overview.png)
1616

17-
To get started, explore [AI audit](/ai-audit/).
17+
To get started, explore [AI audit](/ai-crawl-control/).
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
title: Introducing Pay Per Crawl (private beta)
3+
description: A new feature which allows site owners to charge crawlers for access to their content
4+
date: 2025-07-01
5+
---
6+
7+
We are introducing a new feature of [AI Crawl Control](/ai-crawl-control/) — Pay Per Crawl. [Pay Per Crawl](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) enables site owners to require payment from AI crawlers every time the crawlers access their content, thereby fostering a fairer Internet by enabling site owners to control and monetize how their content gets used by AI.
8+
9+
![Pay per crawl](~/assets/images/changelog/ai-crawl-control/pay-per-crawl.png)
10+
11+
**For Site Owners:**
12+
13+
- Set pricing and select which crawlers to charge for content access
14+
- Manage payments via Stripe
15+
- Monitor analytics on successful content deliveries
16+
17+
**For AI Crawler Owners:**
18+
19+
- Use HTTP headers to request and accept pricing
20+
- Receive clear confirmations on charges for accessed content
21+
22+
Learn more in the [Pay Per Crawl documentation](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/).
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
title: AI Crawl Control refresh
3+
description: More intuitive AI crawler management
4+
date: 2025-07-01
5+
---
6+
7+
We redesigned the AI Crawl Control dashboard to provide more intuitive and granular control over AI crawlers.
8+
9+
- From the new **AI Crawlers** tab: block specific AI crawlers.
10+
- From the new **Metrics** tab: view AI Crawl Control metrics.
11+
12+
![Block AI crawlers](~/assets/images/changelog/ai-crawl-control/manage-ai-crawlers.png)
13+
14+
![Analyze AI crawler activity](~/assets/images/changelog/ai-crawl-control/analyze-metrics.png)
15+
16+
To get started, explore:
17+
18+
- [Manage AI crawlers](/ai-crawl-control/features/manage-ai-crawlers/).
19+
- [Analyze AI traffic](/ai-crawl-control/features/analyze-ai-traffic/).

src/content/docs/ai-audit/changelog.mdx

Lines changed: 0 additions & 12 deletions
This file was deleted.
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
pcx_content_type: changelog
3+
title: Changelog
4+
sidebar:
5+
order: 100
6+
---
7+
8+
import { ProductChangelog } from "~/components";
9+
10+
{/* <!-- Actual content lives in /src/content/changelog/ai-crawl-control/. Update the file there for new entries to appear here. For more details, refer to https://developers.cloudflare.com/style-guide/documentation-content-strategy/content-types/changelog/#yaml-file --> */}
11+
12+
<ProductChangelog product="ai-crawl-control" />

0 commit comments

Comments
 (0)