You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* AI Audit update docs
* AI bots -> crawler audit
* Updating main landing page
* Fixing bad links
* Adding changelog entry
* Update src/content/changelog/ai-audit/2025-06-17-refresh.mdx
Co-authored-by: Kody Jackson <[email protected]>
* Referring to Crawlers instead of bot for consistency
* Adding a release note entry
* Removing incorrect update
* Update src/content/changelog/ai-audit/2025-06-17-refresh.mdx
* Final alignment changes
* Adding a note about filtering the table.
* Updating the filename to match date of release
* Updating changelog to match latest PRD
* Updating filters per plan
* Using SyncKey
* Minor tweak
* Last min content audit
---------
Co-authored-by: Kody Jackson <[email protected]>
AI Audit enables you to take specific action for each AI crawler.
11
+
12
+
To manage AI crawlers:
13
+
14
+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
15
+
2. Go to **AI Audit**.
16
+
3. Go to the **AI Crawlers** tab.
17
+
18
+
## View the list of AI crawlers
19
+
20
+
The **AI Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information, depending on the type of plan you are on.
| Operator | The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers. |
37
+
| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/). |
38
+
| Requests | The number of times the AI crawler has requested to crawl your content. |
39
+
| Robots.txt violations | The number of times the AI crawler has violated your <GlossaryTooltipterm="robots.txt">`robots.txt`</GlossaryTooltip> file. |
40
+
| Block | Toggle for blocking specific AI crawlers. |
41
+
42
+
</TabItem>
43
+
</Tabs>
44
+
45
+
## Block access
46
+
47
+
You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
48
+
49
+
Use the toggles in the **Block** column to block specific AI crawlers from accessing your content.
50
+
51
+
## Filter AI crawler data
52
+
53
+
You can use filters to narrow the scope of your result.
54
+
55
+
-**Name:** Search the name of the AI crawler.
56
+
-**Operator:** Filter by the AI crawler operator.
57
+
-**Category:** Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).
58
+
59
+
The values of the table will update according to your filter.
- View AI crawlers that are interacting with pages in your domain (a [Cloudflare zone](/fundamentals/concepts/accounts-and-zones/#zones)).
19
-
-Create a rule to block AI crawlers on your pages.
19
+
-Use AI Audit to block individual crawlers from accessing your content.
20
20
21
21
## Prerequisites
22
22
23
23
1. Sign up for a [Cloudflare account](https://dash.cloudflare.com/sign-up/).
24
24
2.[Connect your domain to Cloudflare](/fundamentals/manage-domains/add-site/).
25
25
3. Make sure your domain is [proxying traffic through Cloudflare](/fundamentals/concepts/how-cloudflare-works/#cloudflare-as-a-reverse-proxy).
26
26
27
-
## 1. Block all AI crawlers
27
+
## 1. Block specific AI crawlers
28
28
29
-
To use AI Audit:
29
+
<TabssyncKey="plans">
30
+
<TabItemlabel="Free plan">
31
+
32
+
To only block specific AI crawlers:
30
33
31
34
{/* prettier-ignore */}
32
35
<Steps>
33
-
1.Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
34
-
2.Go to **AI Audit**.
35
-
3.From **Most Popular Paths**, select **Block All**.
36
-
4. From the **Bot traffic**page, under **Block AI Bots**, select **Enable**.
36
+
1.Go to **AI Audit**.
37
+
2.From **AI Crawlers** tab, go to the list of AI crawlers.
38
+
3.Review the AI crawlers that are accessing your domain.
39
+
4. From the **Block**column, toggle the switch for AI crawlers you with to block.
37
40
</Steps>
38
41
39
-
This feature also enables Cloudflare's [Bot Fight Mode: Block AI bots](/bots/get-started/bot-fight-mode/#block-ai-bots).
40
-
41
-
You can also create more complex rules when taking action on AI crawlers. For more information on creating more specific rules, refer to [Create a custom rule in the dashboard
42
-
](/waf/custom-rules/create-dashboard/).
43
-
44
-
<Detailsheader="Which bots will Cloudflare block?">
45
-
<Renderfile="list-ai-bots"product="bots" />
46
-
</Details>
47
-
48
-
:::note
49
-
50
-
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
42
+
:::note[Quality of AI crawler detection]
43
+
On the free plan, AI Audit identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Audit to detect easy-to-detect (well-known) AI crawlers.
51
44
45
+
Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field.
52
46
:::
53
47
54
-
## 2. Block specific bot categories (Enterprise plan only)
48
+
</TabItem> <TabItemlabel="Paid plans">
55
49
56
-
Customers on the Enterprise plan -- and with a [Bot Management subscription](/bots/plans/bm-subscription/) -- can choose to only block specific AI crawlers, while allowing others.
50
+
To only block specific AI crawlers:
57
51
58
52
{/* prettier-ignore */}
59
53
<Steps>
60
-
1. Go to the **AI Audit**.
61
-
2. From **Most Popular Paths**, select **Block Some**.
62
-
3. From the **Security rules** page, select **Create rule** > **Custom rules**.
63
-
4. Provide a name for the custom rule. For example, "Block unwanted AI crawlers".
64
-
5. From the **Field** dropdown, select **Verified Bot Category**.
65
-
6. From the **Value** dropdown, select the specific bot category you wish to block.
66
-
- You can use **And** / **Or** buttons to add additional conditions. For example, you can use multiple **Or** options to include multiple bot categories in the same rule.
2. From **AI Crawlers** tab, go to the list of AI crawlers.
56
+
3. Review key information such as:
57
+
- Bot operator
58
+
- Number of requests sent by the AI crawler
59
+
- Whether the AI crawler is allowed according to your `robots.txt`, or the number of times the AI crawler has violated your `robots.txt`.
60
+
4. From the **Block** column, toggle the switch for AI crawlers you with to block.
73
61
</Steps>
74
62
75
-
This custom rule will only block the AI bots which belong to the [verified bot categories](/bots/concepts/bot/verified-bots/categories/) you have included in your rule (in step 6).
63
+
</TabItem>
64
+
</Tabs>
76
65
77
-
For more information on creating a custom WAF rule, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
66
+
For more information, refer to [Manage AI crawlers](/ai-audit/features/manage-ai-crawlers/).
78
67
79
-
## 3. Review detected AI crawlers
68
+
You can also create more complex rules when taking action on AI crawlers, using [Cloudflare WAF](/waf/). For more information on creating more specific rules, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
80
69
81
-
Review the AI crawlers detected on your site in the **Metrics** tab of the Cloudflare dashboard for key metrics.
70
+
## 2. Review detected AI crawlers
71
+
72
+
<Tabs>
73
+
<TabItemlabel="All plans">
74
+
75
+
Review the AI crawlers detected on your site.
76
+
77
+
<Steps>
78
+
1. Go to **AI Audit**.
79
+
2. From the **Metrics** tab, view key metrics on AI crawlers.
80
+
</Steps>
81
+
</TabItem>
82
+
</Tabs>
82
83
83
-
Refer to [Detect AI crawlers](/ai-audit/features/detect-ai-crawlers/)for more information.
84
+
Note that on free plans, the **Metrics** tab only displays metrics for the past 24 hours.
0 commit comments