Skip to content

Commit 281b4ff

Browse files
OxyjunOxyjun
committed
AI Audit update docs
1 parent 18c389f commit 281b4ff

File tree

9 files changed

+206
-111
lines changed

9 files changed

+206
-111
lines changed

‎public/__redirects‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,10 @@
138138
# ai
139139
/ai/ /use-cases/ai/ 301
140140

141+
# ai-audit
142+
/ai-audit/features/detect-ai-crawlers/ /ai-audit/features/analyze-ai-crawlers/ 301
143+
/ai-audit/features/enforce-robots-txt/ /ai-audit/features/analyze-ai-crawlers/ 301
144+
141145
# AI Gateway
142146
/ai-gateway/get-started/configuring-settings/ /ai-gateway/get-started/ 301
143147
/ai-gateway/get-started/connecting-applications/ /ai-gateway/get-started/ 301

‎src/content/docs/ai-audit/features/analyze-ai-crawlers.mdx‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
title: Analyze AI crawlers
3+
pcx_content_type: concept
4+
sidebar:
5+
order: 7
6+
---
7+
8+
:::note[AI Audit metrics]
9+
AI Audit metrics are only available for customers on a paid plan.
10+
:::
11+
12+
AI Audit metrics provides you with key insights on how AI crawlers are interacting with your website.
13+
14+
To analyze AI crawlers:
15+
16+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
17+
2. Go to **AI Audit**.
18+
3. Go to the **Metrics** tab.
19+
20+
## View AI Audit metrics
21+
22+
AI Audit provides you with the following metrics to help you understand how AI crawlers are interacting with your website.
23+
24+
| Metric | Description |
25+
| ---------------------------------- | ------------------------------------------------------------------------ |
26+
| Total requests | The total number of requests to crawl your website, from all AI crawlers |
27+
| Allowed requests | The number of requests you have allowed (by allowing AI crawlers) |
28+
| Blocked requests | The number of requests you have blocked (by blocking AI crawlers) |
29+
| Requests by AI crawlers | A graph which displays the number of crawl requests from each AI crawler |
30+
| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler |
31+
| Hosts | Top five most requested hosts (website domains) |
32+
| Paths | Top five most requested paths |
33+
| Bot Operators | Top five most active bot operators (by requests) |
34+
35+
## Filter the date range
36+
37+
You can use date filter to narrow the scope of your result.
38+
39+
- **Date range:** Filter the date range of your results. You can choose from three predetermined date ranges:
40+
- Past 7 days
41+
- Past 14 days
42+
- Past month
43+
44+
The values of the AI Audit metrics will update according to your filter.

‎src/content/docs/ai-audit/features/detect-ai-crawlers.mdx‎

Lines changed: 0 additions & 46 deletions
This file was deleted.

‎src/content/docs/ai-audit/features/enforce-robots-txt.mdx‎

Lines changed: 0 additions & 37 deletions
This file was deleted.

‎src/content/docs/ai-audit/features/manage-ai-crawlers.mdx‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
---
2+
title: Manage AI crawlers
3+
pcx_content_type: concept
4+
sidebar:
5+
order: 2
6+
---
7+
8+
import { Steps, GlossaryTooltip, Tabs, TabItem } from "~/components";
9+
10+
AI Audit enables you to take specific action for each AI crawler.
11+
12+
To manage AI crawlers:
13+
14+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
15+
2. Go to **AI Audit**.
16+
3. Go to the **AI Crawlers** tab.
17+
18+
## View the list of AI crawlers
19+
20+
The **AI Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information, depending on the type of plan you are on.
21+
22+
<Tabs>
23+
<TabItem label="Free plan">
24+
| Column | Details |
25+
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
26+
| AI Crawlers | The name of the AI crawler. |
27+
| Operator | The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers. |
28+
| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/). |
29+
| Block | Toggle for blocking specific AI crawlers. |
30+
31+
</TabItem> <TabItem label="Paid plan">
32+
33+
| Column | Details |
34+
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
35+
| AI Crawlers | The name of the AI crawler. |
36+
| Operator | The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers. |
37+
| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/). |
38+
| Requests | The number of times the AI crawler has requested to crawl your content. |
39+
| Robots.txt violations | The number of times the AI crawler has violated your <GlossaryTooltip term="robots.txt">`robots.txt`</GlossaryTooltip> file. |
40+
| Block | Toggle for blocking specific AI crawlers. |
41+
42+
</TabItem>
43+
</Tabs>
44+
45+
## Block access
46+
47+
You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
48+
49+
## Filter AI crawler data
50+
51+
You can use filters to narrow the scope of your result.
52+
53+
- **Name:** Filter by the bot name by searching the name of the bot.
54+
- **Operator:** Filter by the AI crawler operator.
55+
- **Category:** Filter by the category of the AI bot (for example, AI crawler, AI assistant, or archiver).
56+
- **Date range:** Filter the date range of your results. You can choose from three predetermined date ranges:
57+
- Past 7 days
58+
- Past 14 days
59+
- Past month
60+
61+
The values of the AI Audit metrics will update according to your filter.
62+
63+
## Related resources
64+
65+
- Use [Pay per Crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) to charge AI crawlers every time they access your content.

‎src/content/docs/ai-audit/get-started.mdx‎

Lines changed: 60 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ head:
1111
description: Learn how to set up AI Audit.
1212
---
1313

14-
import { Details, Render, Steps } from "~/components";
14+
import { Details, Render, Steps, Tabs, TabItem, GlossaryTooltip } from "~/components";
1515

1616
This guide instructs you on how to:
1717

@@ -26,61 +26,93 @@ This guide instructs you on how to:
2626

2727
## 1. Block all AI crawlers
2828

29-
To use AI Audit:
29+
<Tabs>
30+
<TabItem label="All plans">
31+
To block all AI crawlers:
3032

3133
{/* prettier-ignore */}
3234
<Steps>
3335
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
3436
2. Go to **AI Audit**.
35-
3. From **Most Popular Paths**, select **Block All**.
36-
4. From the **Bot traffic** page, under **Block AI Bots**, select **Enable**.
37+
3. Go to the **AI Crawlers** tab.
38+
4. From **Block AI Bots**, toggle the switch.
3739
</Steps>
3840

39-
This feature also enables Cloudflare's [Bot Fight Mode: Block AI bots](/bots/get-started/bot-fight-mode/#block-ai-bots).
41+
This blocks all AI crawlers requesting access to your domain. You will also see all of the individual switches for bots automatically toggle to **Block**.
4042

41-
You can also create more complex rules when taking action on AI crawlers. For more information on creating more specific rules, refer to [Create a custom rule in the dashboard
42-
](/waf/custom-rules/create-dashboard/).
43+
Note that blocking all AI crawlers using this method also enables Cloudflare's [Bot Fight Mode: Block AI bots](/bots/get-started/bot-fight-mode/#block-ai-bots).
4344

4445
<Details header="Which bots will Cloudflare block?">
4546
<Render file="list-ai-bots" product="bots" />
4647
</Details>
4748

4849
:::note
49-
5050
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
51+
:::
52+
</TabItem>
53+
</Tabs>
54+
55+
## 2. Block specific AI crawlers
56+
57+
<Tabs syncKey="plans">
58+
<TabItem label="Free plan">
59+
60+
To only block specific AI crawlers:
61+
62+
{/* prettier-ignore */}
63+
<Steps>
64+
1. Go to **AI Audit**.
65+
2. From **AI Crawlers** tab, go to the list of AI crawlers.
66+
3. Review the AI crawlers that are accessing your domain.
67+
4. From the **Block** column, toggle the switch for AI crawlers you with to block.
68+
</Steps>
69+
70+
:::note[Quality of AI crawler detection]
71+
On the free plan, AI Audit identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Audit to detect easy-to-detect (well-known) AI crawlers.
5172

73+
Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field.
5274
:::
5375

54-
## 2. Block specific bot categories (Enterprise plan only)
76+
</TabItem> <TabItem label="Paid plan">
5577

56-
Customers on the Enterprise plan -- and with a [Bot Management subscription](/bots/plans/bm-subscription/) -- can choose to only block specific AI crawlers, while allowing others.
78+
To only block specific AI crawlers:
5779

5880
{/* prettier-ignore */}
5981
<Steps>
60-
1. Go to the **AI Audit**.
61-
2. From **Most Popular Paths**, select **Block Some**.
62-
3. From the **Security rules** page, select **Create rule** > **Custom rules**.
63-
4. Provide a name for the custom rule. For example, "Block unwanted AI crawlers".
64-
5. From the **Field** dropdown, select **Verified Bot Category**.
65-
6. From the **Value** dropdown, select the specific bot category you wish to block.
66-
- You can use **And** / **Or** buttons to add additional conditions. For example, you can use multiple **Or** options to include multiple bot categories in the same rule.
67-
7. From the **Then take action...** section:
68-
- For **Choose action**, select **Block**.
69-
- For **With response type**, select **Default Cloudflare WAF block page**.
70-
8. From the **Place at** section:
71-
- For **Select order**, select **First**.
72-
9. Select **Save**.
82+
1. Go to **AI Audit**.
83+
2. From **AI Crawlers** tab, go to the list of AI crawlers.
84+
3. Review key information such as:
85+
- Bot operator
86+
- Number of requests sent by the AI crawler
87+
- Whether the bot is allowed according to your `robots.txt`, or the number of times the AI crawler has violated your `robots.txt`.
88+
4. From the **Block** column, toggle the switch for AI crawlers you with to block.
7389
</Steps>
7490

75-
This custom rule will only block the AI bots which belong to the [verified bot categories](/bots/concepts/bot/verified-bots/categories/) you have included in your rule (in step 6).
91+
</TabItem>
92+
</Tabs>
93+
94+
For more information, refer to [Manage AI crawlers](/ai-audit/features/manage-ai-crawlers/).
95+
96+
You can also create more complex rules when taking action on AI crawlers, using [Cloudflare WAF](/waf/). For more information on creating more specific rules, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
7697

77-
For more information on creating a custom WAF rule, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
98+
## 3. Review detected AI crawlers (Enterprise plan only)
7899

79-
## 3. Review detected AI crawlers
100+
<Tabs syncKey="plans">
101+
<TabItem label="Free plan">
102+
AI Audit metrics are only available on paid plans. Upgrade your plan to view key metrics on AI crawlers.
80103

81-
Review the AI crawlers detected on your site in the **Metrics** tab of the Cloudflare dashboard for key metrics.
104+
</TabItem><TabItem label="Paid plan">
105+
106+
Review the AI crawlers detected on your site.
107+
108+
<Steps>
109+
1. Go to **AI Audit**.
110+
2. From the **Metrics** tab, view key metrics on AI crawlers.
111+
</Steps>
112+
</TabItem>
113+
</Tabs>
82114

83-
Refer to [Detect AI crawlers](/ai-audit/features/detect-ai-crawlers/) for more information.
115+
Refer to [Detect AI crawlers](/ai-audit/features/detect-ai-crawlers/) for more information on AI Audit metrics.
84116

85117
## Related resources
86118

‎src/content/docs/ai-audit/reference/glossary.mdx‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
title: Glossary
3+
pcx_content_type: glossary
4+
sidebar:
5+
order: 12
6+
7+
---
8+
9+
import { Glossary } from "~/components"
10+
11+
Review the definitions for terms used across Cloudflare's AI Audit documentation.
12+
13+
<Glossary product="ai-audit" />

‎src/content/docs/ai-audit/reference/index.mdx‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
title: Reference
3+
pcx_content_type: navigation
4+
sidebar:
5+
group:
6+
hideIndex: true
7+
order: 10
8+
---
9+
10+
import { DirectoryListing } from "~/components";
11+
12+
<DirectoryListing />

‎src/content/glossary/ai-audit.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
productName: AI Audit
3+
entries:
4+
- term: robots.txt
5+
general_definition: |-
6+
A text file which lists pages in your website that are off-limits for bots. Well-behaved bots respect this file, but some bots may violate it. You can [enforce robots.txt](/ai-audit/features/enforce-robots-txt/) with Cloudflare WAF custom rules.
7+
8+

0 commit comments

Comments
 (0)