Skip to content
Merged
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@
/src/content/docs/d1/ @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @harshil1712 @cloudflare/pcx-technical-writing
/src/content/release-notes/d1.yaml @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @cloudflare/pcx-technical-writing
/src/content/partials/d1/ @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @harshil1712 @cloudflare/pcx-technical-writing
/src/content/docs/ai-audit/ @oxyjun @kodster28 @cloudflare/pcx-technical-writing
/src/content/docs/durable-objects/ @elithrar @vy-ton @joshthoward @oxyjun @harshil1712 @cloudflare/pcx-technical-writing
/src/content/release-notes/durable-objects.yaml @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @cloudflare/pcx-technical-writing
/src/content/docs/email-routing/ @cloudflare/pcx-technical-writing
Expand Down
29 changes: 29 additions & 0 deletions src/content/docs/ai-audit/features/ai-crawler-detection.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: AI crawler detection
pcx_content_type: concept
sidebar:
order: 2
---

AI Audit metrics provides you with insight on how AI crawlers are interacting with your website.

## View AI Audit metrics

AI Audit provides you with the following metrics to help you understand how AI crawlers are interacting with your website.

| Metric | Description |
| ------------------------------------- | ----------------------------------------------------------------------------- |
| Total AI crawler requests | The total number of requests to crawl your website, from all AI crawlers |
| Blocked requests | The number of requests you have blocked (by blocking AI crawlers) |
| Allowed requests | The number of requests you have allowed (by allowing AI crawlers) |
| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler |
| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler |
| Hosts | The owner of the AI crawler |
| Overall popular paths | The most popular pages crawled by AI crawlers, from all AI crawlers |
| Most active AI crawlers by bot owners | The AI crawler owners with the highest number of requests to access your site |

## Filter date range

You can use the date filter to choose the period of time you wish to analyze.

Click on the date filter dropdown and select the range of dates. The values of the AI Audit metrics will update according to your filter.
12 changes: 12 additions & 0 deletions src/content/docs/ai-audit/features/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
title: Features
pcx_content_type: navigation
sidebar:
group:
hideIndex: true
order: 5
---

import { DirectoryListing } from "~/components";

<DirectoryListing />
46 changes: 46 additions & 0 deletions src/content/docs/ai-audit/get-started.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
title: Get started
pcx_content_type: get-started
sidebar:
order: 2
group:
badge: beta
head:
- tag: title
content: Get started with Cloudflare AI Audit
description: Learn how to set up AI Audit.

---

import { Render, Steps } from "~/components";

## Prerequisites

1. Sign up for a [Cloudflare account](https://dash.cloudflare.com/sign-up/).
2. [Connect your domain to Cloudflare](/fundamentals/manage-domains/add-site/).

## 1. Block all AI crawlers

To enable AI Audit:

<Steps>
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
2. Go to **AI Audit**.
3. From **AI Crawlers** tab, toggle **Block AI Crawlers**.
4. From the **Bot traffic** page, under **Block AI Bots**, select **Enable**.
</Steps>

## 2. Block specific AI crawlers (Enterprise plan only)

Customers on the Enterprise plan can choose to only block specific AI crawlers, while allowing others.

<Steps>
1. Go to the **AI Crawlers** tab.
2. Toggle the switch under the **Block** column for the AI crawlers you wish to block.
</Steps>

## 3. Review detected AI crawlers

Review the AI crawlers detected on your site in the **Metrics** tab of the Cloudflare dashboard for key metrics.

Refer to [AI crawler detection](/ai-audit/ai-crawler-detection/) for more information.
42 changes: 42 additions & 0 deletions src/content/docs/ai-audit/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: AI Audit
pcx_content_type: overview
sidebar:
order: 1
badge: Beta
head:
- tag: title
content: Overview
description: AI Audit is a tool which allows you to analyze and control how third-party AI crawlers interact with your website.
---

import { Description, Feature, FeatureTable, Plan, LinkButton } from "~/components";

<Description>

Analyze and control third-party AI crawlers in your website.

</Description>

<Plan type="all" />

AI Audit helps manage AI crawlers on your website by providing visibility on which crawlers are accessing your webpage, and allowing you to block unwanted crawlers.

<LinkButton href="/ai-audit/get-started/">Get started </LinkButton>

:::note[Beta phase]
AI Audit is currently only available as a beta product.
:::

---

## Features

<Feature
header="AI crawler detection"
href="/ai-audit/features/ai-crawler-detection/"
cta="View AI crawlers"
>
Displays information about AI crawlers in your domains' pages.
</Feature>

8 changes: 8 additions & 0 deletions src/content/docs/ai-audit/reference/allowed-ai-crawlers.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
title: List of allowed AI crawlers
pcx_content_type: concept
sidebar:
order: 2
---

TBC
12 changes: 12 additions & 0 deletions src/content/docs/ai-audit/reference/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
title: Reference
pcx_content_type: navigation
sidebar:
group:
hideIndex: true
order: 10
---

import { DirectoryListing } from "~/components";

<DirectoryListing />
14 changes: 14 additions & 0 deletions src/content/docs/ai-audit/release-notes.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
pcx_content_type: changelog
title: Release notes
release_notes_file_name:
- ai-audit
sidebar:
order: 100
---

import { ProductReleaseNotes } from "~/components";

{/* <!-- Actual content lives in /src/content/release-notes/ai-audit.yaml. Update the file there for new entries to appear here. For more details, refer to https://developers.cloudflare.com/style-guide/documentation-content-strategy/content-types/changelog/#yaml-file --> */}

<ProductReleaseNotes />
9 changes: 9 additions & 0 deletions src/content/products/ai-audit.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
name: AI Audit
product:
title: AI Audit
url: /ai-audit/
group: Core platform
meta:
title: AI Audit
description: Analyze and control third-party AI crawlers in your website
author: "@cloudflare"
11 changes: 11 additions & 0 deletions src/content/release-notes/ai-audit.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
link: "/ai-audit/changelog/"
productName: AI Audit
productLink: "/ai-audit/"
productArea: Core platform
productAreaLink: /fundamentals/reference/changelog/platform/
entries:
- publish_date: "2025-06-09"
title: Documentation for AI Audit
description: There is now documentation for AI Audit

1 change: 1 addition & 0 deletions src/icons/ai-audit.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading