Skip to content

Commit 942993f

Browse files
Oxyjunkodster28
Oxyjun
and
kodster28
authored
[AI Audit] Initialising AI Audit docs (#22950)
* Initialising AI Audit docs * Fixing product yaml file. * Setting up more pages for AI Audit docs * Documenting AI Audit metrics * Iterating over feedback part 1 * changelog update * proxy clarification * Adding more content, matching text to current UI. * Adding docs on Violations only toggle. * Removing inaccuracy. * Wording clarification. * Fixing incorrect sentence * minor tweaks --------- Co-authored-by: kodster28 <[email protected]>
1 parent 16db58f commit 942993f

File tree

12 files changed

+292
-0
lines changed

12 files changed

+292
-0
lines changed

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@
9999
/src/content/docs/d1/ @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @harshil1712 @cloudflare/pcx-technical-writing
100100
/src/content/release-notes/d1.yaml @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @cloudflare/pcx-technical-writing
101101
/src/content/partials/d1/ @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @harshil1712 @cloudflare/pcx-technical-writing
102+
/src/content/docs/ai-audit/ @oxyjun @kodster28 @cloudflare/pcx-technical-writing
102103
/src/content/docs/durable-objects/ @elithrar @vy-ton @joshthoward @oxyjun @harshil1712 @cloudflare/pcx-technical-writing
103104
/src/content/release-notes/durable-objects.yaml @elithrar @rozenmd @vy-ton @joshthoward @oxyjun @cloudflare/pcx-technical-writing
104105
/src/content/docs/email-routing/ @cloudflare/pcx-technical-writing

‎src/assets/images/changelog/ai-audit/ai-audit-overview.png‎

109 KB
Loading

‎src/content/changelog/ai-audit/2024-09-23-ai-audit-launch.mdx‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
title: AI Audit
3+
description: AI Audit is available to all customers
4+
date: 2024-09-23T11:00:00Z
5+
---
6+
7+
Every site on Cloudflare now has access to [**AI Audit**](/ai-audit/), which summarizes the crawling behavior of popular and known AI services.
8+
9+
You can use this data to:
10+
11+
- Understand how and how often crawlers access your site (and which content is the most popular).
12+
- Block some or all of the AI bots accessing your site.
13+
- Use Cloudflare to enforce your `robots.txt` policy via an automatic WAF rule.
14+
15+
![View AI bot activity with AI Audit](~/assets/images/changelog/ai-audit/ai-audit-overview.png)
16+
17+
To get started, explore [AI audit](/ai-audit/).

‎src/content/docs/ai-audit/changelog.mdx‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
pcx_content_type: changelog
3+
title: Changelog
4+
sidebar:
5+
order: 100
6+
---
7+
8+
import { ProductChangelog } from "~/components";
9+
10+
{/* <!-- Actual content lives in /src/content/changelog/ai-audit/. Update the file there for new entries to appear here. For more details, refer to https://developers.cloudflare.com/style-guide/documentation-content-strategy/content-types/changelog/#yaml-file --> */}
11+
12+
<ProductChangelog product="ai-audit" />

‎src/content/docs/ai-audit/features/detect-ai-crawlers.mdx‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
title: Detect AI crawlers
3+
pcx_content_type: concept
4+
sidebar:
5+
order: 2
6+
---
7+
8+
AI Audit metrics provides you with insight on how AI crawlers are interacting with your website.
9+
10+
## View AI Audit metrics
11+
12+
AI Audit provides you with the following metrics to help you understand how AI crawlers are interacting with your website.
13+
14+
| Metric | Description |
15+
| --------------------------------- | ------------------------------------------------------------------------ |
16+
| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler |
17+
| Summary | A list of AI crawlers with the most number of crawl requests |
18+
| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler |
19+
20+
The **Summary** table also enables you to [Enforce your robots.txt](/ai-audit/features/enforce-robots-txt/).
21+
22+
## Filter AI crawler data
23+
24+
You can use filters to narrow the scope of your result.
25+
26+
- **Provider:** Filter by the AI crawler owners.
27+
- **Bot type:** Filter by the type of the AI bot (for example, AI crawler, AI assistant, or archiver).
28+
- **Date range:** Filter the date range of your results. You can choose from three predetermined date ranges:
29+
- Past 7 days
30+
- Past 14 days
31+
- Past month
32+
33+
The values of the AI Audit metrics will update according to your filter.
34+
35+
## Filter subdomains
36+
37+
You can use the subdomain filter to narrow the scope of your result.
38+
39+
From the dropdown, select either **All subdomains**, or the specific subdomain you wish to view.
40+
41+
Selecting a specific subdomain allows you to access:
42+
43+
- **Violations only** toggle: Toggles the AI Audit page to only display bots which are violating your configured rules.
44+
- [**Enforce robots.txt policy**](/ai-audit/features/enforce-robots-txt/): Ensure bots cannot access webpages which are off-limits, as specified in your `robots.txt` file.
45+
46+
The values of the AI Audit metrics will update according to your filter.

‎src/content/docs/ai-audit/features/enforce-robots-txt.mdx‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
title: Enforce robots.txt
3+
pcx_content_type: concept
4+
sidebar:
5+
order: 5
6+
---
7+
8+
import { Steps } from "~/components";
9+
10+
AI Audit allows you to enforce [`robots.txt`](/radar/glossary/#robotstxt) which instructs bots which webpages they can and cannot access.
11+
12+
To enforce `robots.txt`:
13+
14+
<Steps>
15+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
16+
2. Go to **AI Audit**.
17+
3. From the dropdown at the top of the page, select a specific subdomain where you wish you enforce `robots.txt`.
18+
4. From **Summary**, select **Enforce robots.txt policy**.
19+
5. From the **Enforce your robots.txt policy** page, select **Go to WAF custom rules**.
20+
6. From the **New custom rule** page, name your custom rule.
21+
- The page will automatically populate the values for the custom rule.
22+
7. From **Then take action...**:
23+
- For **Choose action**, select **Block**.
24+
- For **With response type**, select **Default Cloudflare WAF block page**.
25+
8. From **Place at**:
26+
- For **Select order**, select **Last**.
27+
9. Select **Deploy**.
28+
</Steps>
29+
30+
This custom rule ensures that bots cannot access the pages specified in your `robots.txt` file.
31+
32+
## Related resources
33+
34+
For more information, refer to the following resources.
35+
36+
- [What is robots.txt? | How a robots.txt file works](https://www.cloudflare.com/en-gb/learning/bots/what-is-robots-txt/)
37+
- [Direct AI crawlers with managed robots.txt](/bots/additional-configurations/managed-robots-txt/)

‎src/content/docs/ai-audit/features/index.mdx‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
title: Features
3+
pcx_content_type: navigation
4+
sidebar:
5+
group:
6+
hideIndex: true
7+
order: 5
8+
---
9+
10+
import { DirectoryListing } from "~/components";
11+
12+
<DirectoryListing />

‎src/content/docs/ai-audit/get-started.mdx‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
---
2+
title: Get started
3+
pcx_content_type: get-started
4+
sidebar:
5+
order: 2
6+
group:
7+
badge: beta
8+
head:
9+
- tag: title
10+
content: Get started with Cloudflare AI Audit
11+
description: Learn how to set up AI Audit.
12+
---
13+
14+
import { Render, Steps } from "~/components";
15+
16+
This guide instructs you through
17+
18+
- Viewing AI crawlers that are interacting with your domain.
19+
- Creating a rule to block AI crawlers on your website.
20+
21+
## Prerequisites
22+
23+
1. Sign up for a [Cloudflare account](https://dash.cloudflare.com/sign-up/).
24+
2. [Connect your domain to Cloudflare](/fundamentals/manage-domains/add-site/).
25+
3. Make sure your domain is [proxying traffic through Cloudflare](/fundamentals/concepts/how-cloudflare-works/#cloudflare-as-a-reverse-proxy).
26+
27+
## 1. Block all AI crawlers
28+
29+
To use AI Audit:
30+
31+
{/* prettier-ignore */}
32+
<Steps>
33+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
34+
2. Go to **AI Audit**.
35+
3. From **Most Popular Paths**, select **Block All**.
36+
4. From the **Bot traffic** page, under **Block AI Bots**, select **Enable**.
37+
</Steps>
38+
39+
## 2. Block specific bot categories (Enterprise plan only)
40+
41+
Customers on the Enterprise plan -- and with a [Bot Management subscription](/bots/plans/bm-subscription/) -- can choose to only block specific AI crawlers, while allowing others.
42+
43+
{/* prettier-ignore */}
44+
<Steps>
45+
1. Go to the **AI Audit**.
46+
2. From **Most Popular Paths**, select **Block Some**.
47+
3. From the **Security rules** page, select **Create rule** > **Custom rules**.
48+
4. Provide a name for the custom rule. For example, "Block unwanted AI crawlers".
49+
5. From the **Field** dropdown, select **Verified Bot Category**.
50+
6. From the **Value** dropdown, select the specific bot category you wish to block.
51+
- You can use **And** / **Or** buttons to add additional conditions. For example, you can use multiple **Or** options to include multiple bot categories in the same rule.
52+
7. From the **Then take action...** section:
53+
- For **Choose action**, select **Block**.
54+
- For **With response type**, select **Default Cloudflare WAF block page**.
55+
8. From the **Place at** section:
56+
- For **Select order**, select **First**.
57+
9. Select **Save**.
58+
</Steps>
59+
60+
This custom rule will only block the AI bots which belong to the [verified bot categories](/bots/concepts/bot/verified-bots/categories/) you have included in your rule (in step 6).
61+
62+
For more information on creating a custom WAF rule, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
63+
64+
## 3. Review detected AI crawlers
65+
66+
Review the AI crawlers detected on your site in the **Metrics** tab of the Cloudflare dashboard for key metrics.
67+
68+
Refer to [Detect AI crawlers](/ai-audit/features/detect-ai-crawlers/) for more information.
69+
70+
## Related resources
71+
72+
Refer to the following related resources:
73+
74+
- Cloudflare blog: [Start auditing and controlling the AI models accessing your content](https://blog.cloudflare.com/nl-nl/cloudflare-ai-audit-control-ai-content-crawlers/)
75+
- Block AI crawlers that do not adhere to recommended guidelines using [Cloudflare AI Labyrinth](/bots/additional-configurations/ai-labyrinth/).
76+
- [Direct AI crawlers with managed robots.txt](/bots/additional-configurations/managed-robots-txt/).

‎src/content/docs/ai-audit/index.mdx‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
---
2+
title: AI Audit
3+
pcx_content_type: overview
4+
sidebar:
5+
order: 1
6+
badge: Beta
7+
head:
8+
- tag: title
9+
content: Overview
10+
description: AI Audit is a tool which allows you to analyze and control how third-party AI crawlers interact with your website.
11+
---
12+
13+
import { Description, Feature, FeatureTable, Plan, LinkButton, RelatedProduct } from "~/components";
14+
15+
<Description>
16+
17+
Analyze and control third-party AI crawlers in your website.
18+
19+
</Description>
20+
21+
<Plan type="all" />
22+
23+
AI Audit helps manage AI crawlers on your website by providing visibility on which crawlers are accessing your webpage, and allowing you to block unwanted crawlers.
24+
25+
<LinkButton href="/ai-audit/get-started/">Get started </LinkButton>
26+
27+
:::note[Beta phase]
28+
AI Audit is currently only available as a beta product.
29+
:::
30+
31+
---
32+
33+
## Features
34+
35+
<Feature
36+
header="AI crawler detection"
37+
href="/ai-audit/features/ai-crawler-detection/"
38+
cta="View AI crawlers"
39+
>
40+
Displays information about AI crawlers in your domains' pages.
41+
</Feature>
42+
43+
<Feature
44+
header="Enforce robots.txt"
45+
href="/ai-audit/features/enforce-robots-txt/"
46+
cta="Enforce your robots.txt"
47+
>
48+
Enforce your `robots.txt` with a Cloudflare WAF rule.
49+
</Feature>
50+
51+
---
52+
53+
## Related Products
54+
55+
<RelatedProduct
56+
header="Bots"
57+
href="/bots/"
58+
product="bots"
59+
>
60+
Identify and mitigate automated traffic to protect your domain from bad bots.
61+
</RelatedProduct>
62+
63+
<RelatedProduct
64+
header="Web Application Firewall"
65+
href="/waf/"
66+
product="waf"
67+
>
68+
Get automatic protection from vulnerabilities and the flexibility to create custom rules.
69+
</RelatedProduct>

‎src/content/products/ai-audit.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
name: AI Audit
2+
product:
3+
title: AI Audit
4+
url: /ai-audit/
5+
group: Core platform
6+
additional_groups: [AI]
7+
meta:
8+
title: AI Audit
9+
description: Analyze and control third-party AI crawlers in your website
10+
author: "@cloudflare"

0 commit comments

Comments
 (0)