[AI Audit] AI Audit update docs

public/__redirects

@@ -138,6 +138,10 @@
 # ai
 /ai/ /use-cases/ai/ 301
 
+# ai-audit
+/ai-audit/features/detect-ai-crawlers/ /ai-audit/features/analyze-ai-crawlers/ 301
+/ai-audit/features/enforce-robots-txt/ /ai-audit/features/analyze-ai-crawlers/ 301
+
 # AI Gateway
 /ai-gateway/get-started/configuring-settings/ /ai-gateway/get-started/ 301
 /ai-gateway/get-started/connecting-applications/ /ai-gateway/get-started/ 301

src/content/changelog/ai-audit/2025-06-17-refresh.mdx

@@ -0,0 +1,19 @@
+---
+title: AI Audit refresh
+description: More intuitive AI crawler management
+date: 2025-06-17T11:00:00Z
+---
+
+We redesigned the AI Audit dashboard to provide more intuitive and granular control over AI crawlers.
+
+- From the new **AI Crawlers** tab: block all AI crawlers with a single toggle, or block specific AI crawlers.
+- From the new **Metrics** tab: view AI Audit metrics (on paid plans only).
+
+![Block AI crawlers](~/assets/images/changelog/ai-audit/manage-ai-crawlers.png)
+
+![Analyze AI crawler activity](~/assets/images/changelog/ai-audit/analyze-metrics.png)
+
+To get started, explore:
+
+- [Manage AI crawlers](/ai-audit/features/manage-ai-crawlers/).
+- [Analyze AI crawlers](/ai-audit/features/analyze-ai-crawlers/).

src/content/docs/ai-audit/features/analyze-ai-crawlers.mdx

@@ -0,0 +1,44 @@
+---
+title: Analyze AI crawlers
+pcx_content_type: concept
+sidebar:
+  order: 7
+---
+
+:::note[AI Audit metrics]
+AI Audit metrics are only available for customers on a paid plan.
+:::
+
+AI Audit metrics provides you with key insights on how AI crawlers are interacting with your website.
+
+To analyze AI crawlers:
+
+1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
+2. Go to **AI Audit**.
+3. Go to the **Metrics** tab.
+
+## View AI Audit metrics
+
+AI Audit provides you with the following metrics to help you understand how AI crawlers are interacting with your website.
+
+| Metric                             | Description                                                              |
+| ---------------------------------- | ------------------------------------------------------------------------ |
+| Total requests                     | The total number of requests to crawl your website, from all AI crawlers |
+| Allowed requests                   | The number of requests you have allowed (by allowing AI crawlers)        |
+| Blocked requests                   | The number of requests you have blocked (by blocking AI crawlers)        |
+| Requests by AI crawlers            | A graph which displays the number of crawl requests from each AI crawler |
+| Most popular paths by AI crawlers  | The most popular pages crawled by AI crawlers, for each AI crawler       |
+| Hosts                              | Top five most requested hosts (website domains)                          |
+| Paths                              | Top five most requested paths                                            |
+| AI crawler operators               | Top five most active AI crawler operators (by requests)                  |
+
+## Filter the date range
+
+You can use date filter to narrow the scope of your result.
+
+- **Date range:** Filter the date range of your results. You can choose from three predetermined date ranges:
+  - Past 7 days
+  - Past 14 days
+  - Past month
+
+The values of the AI Audit metrics will update according to your filter.

src/content/docs/ai-audit/features/manage-ai-crawlers.mdx

@@ -0,0 +1,61 @@
+---
+title: Manage AI crawlers
+pcx_content_type: concept
+sidebar:
+  order: 2
+---
+
+import { Steps, GlossaryTooltip, Tabs, TabItem } from "~/components";
+
+AI Audit enables you to take specific action for each AI crawler.
+
+To manage AI crawlers:
+
+1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
+2. Go to **AI Audit**.
+3. Go to the **AI Crawlers** tab.
+
+## View the list of AI crawlers
+
+The **AI Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information, depending on the type of plan you are on.
+
+<Tabs>
+<TabItem label="Free plan">
+| Column                | Details                                                                                                                                                             |
+| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| AI Crawlers           | The name of the AI crawler.                                                                                                                                         |
+| Operator              | The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers.                                                                |
+| Category              | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/).                                                   |
+| Block                 | Toggle for blocking specific AI crawlers.                                                                                                                           |
+
+</TabItem> <TabItem label="Paid plan">
+
+| Column                | Details                                                                                                                                                             |
+| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| AI Crawlers           | The name of the AI crawler.                                                                                                                                         |
+| Operator              | The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers.                                                                |
+| Category              | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/).                                                   |
+| Requests              | The number of times the AI crawler has requested to crawl your content.                                                                                             |
+| Robots.txt violations | The number of times the AI crawler has violated your <GlossaryTooltip term="robots.txt">`robots.txt`</GlossaryTooltip> file.                                        |
+| Block                 | Toggle for blocking specific AI crawlers.                                                                                                                           |
+
+</TabItem>
+</Tabs>
+
+## Block access
+
+You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
+
+## Filter AI crawler data
+
+You can use filters to narrow the scope of your result.
+
+- **Name:** Search the name of the AI crawler.
+- **Operator:** Filter by the AI crawler operator.
+- **Category:** Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).
+- **Date range:** Filter the date range of your results. You can choose from three predetermined date ranges:
+  - Past 7 days
+  - Past 14 days
+  - Past month
+
+The values of the AI Audit metrics will update according to your filter.

src/content/docs/ai-audit/get-started.mdx

@@ -11,7 +11,7 @@ head:
 description: Learn how to set up AI Audit.
 ---
 
-import { Details, Render, Steps } from "~/components";
+import { Details, Render, Steps, Tabs, TabItem, GlossaryTooltip } from "~/components";
 
 This guide instructs you on how to:
 
@@ -26,61 +26,93 @@ This guide instructs you on how to:
 
 ## 1. Block all AI crawlers
 
-To use AI Audit:
+<Tabs>
+<TabItem label="All plans">
+To block all AI crawlers:
 
 {/* prettier-ignore */}
 <Steps>
 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
 2. Go to **AI Audit**.
-3. From **Most Popular Paths**, select **Block All**.
-4. From the **Bot traffic** page, under **Block AI Bots**, select **Enable**.
+3. Go to the **AI Crawlers** tab.
+4. From **Block AI Bots**, toggle the switch.
 </Steps>
 
-This feature also enables Cloudflare's [Bot Fight Mode: Block AI bots](/bots/get-started/bot-fight-mode/#block-ai-bots).
+This blocks all AI crawlers requesting access to your domain. You will also see all of the individual switches for AI crawlers automatically toggle to **Block**.
 
-You can also create more complex rules when taking action on AI crawlers. For more information on creating more specific rules, refer to [Create a custom rule in the dashboard
-](/waf/custom-rules/create-dashboard/).
+Note that blocking all AI crawlers using this method also enables Cloudflare's [Bot Fight Mode: Block AI bots](/bots/get-started/bot-fight-mode/#block-ai-bots).
 
 <Details header="Which bots will Cloudflare block?">
 	<Render file="list-ai-bots" product="bots" />
 </Details>
 
 :::note
-
 For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
+:::
+</TabItem>
+</Tabs>
+
+## 2. Block specific AI crawlers
+
+<Tabs syncKey="plans">
+<TabItem label="Free plan">
+
+To only block specific AI crawlers:
+
+{/* prettier-ignore */}
+<Steps>
+1. Go to **AI Audit**.
+2. From **AI Crawlers** tab, go to the list of AI crawlers.
+3. Review the AI crawlers that are accessing your domain.
+4. From the **Block** column, toggle the switch for AI crawlers you with to block.
+</Steps>
+
+:::note[Quality of AI crawler detection]
+On the free plan, AI Audit identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Audit to detect easy-to-detect (well-known) AI crawlers.
 
+Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field.
 :::
 
-## 2. Block specific bot categories (Enterprise plan only)
+</TabItem> <TabItem label="Paid plan">
 
-Customers on the Enterprise plan -- and with a [Bot Management subscription](/bots/plans/bm-subscription/) -- can choose to only block specific AI crawlers, while allowing others.
+To only block specific AI crawlers:
 
 {/* prettier-ignore */}
 <Steps>
-1. Go to the **AI Audit**.
-2. From **Most Popular Paths**, select **Block Some**.
-3. From the **Security rules** page, select **Create rule** > **Custom rules**.
-4. Provide a name for the custom rule. For example, "Block unwanted AI crawlers".
-5. From the **Field** dropdown, select **Verified Bot Category**.
-6. From the **Value** dropdown, select the specific bot category you wish to block.
-   - You can use **And** / **Or** buttons to add additional conditions. For example, you can use multiple **Or** options to include multiple bot categories in the same rule.
-7. From the **Then take action...** section:
-   - For **Choose action**, select **Block**.
-   - For **With response type**, select **Default Cloudflare WAF block page**.
-8. From the **Place at** section:
-   - For **Select order**, select **First**.
-9. Select **Save**.
+1. Go to **AI Audit**.
+2. From **AI Crawlers** tab, go to the list of AI crawlers.
+3. Review key information such as:
+   - Bot operator
+   - Number of requests sent by the AI crawler
+   - Whether the AI crawler is allowed according to your `robots.txt`, or the number of times the AI crawler has violated your `robots.txt`.
+4. From the **Block** column, toggle the switch for AI crawlers you with to block.
 </Steps>
 
-This custom rule will only block the AI bots which belong to the [verified bot categories](/bots/concepts/bot/verified-bots/categories/) you have included in your rule (in step 6).
+</TabItem>
+</Tabs>
+
+For more information, refer to [Manage AI crawlers](/ai-audit/features/manage-ai-crawlers/).
+
+You can also create more complex rules when taking action on AI crawlers, using [Cloudflare WAF](/waf/). For more information on creating more specific rules, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
 
-For more information on creating a custom WAF rule, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
+## 3. Review detected AI crawlers (Enterprise plan only)
 
-## 3. Review detected AI crawlers
+<Tabs syncKey="plans">
+<TabItem label="Free plan">
+AI Audit metrics are only available on paid plans. Upgrade your plan to view key metrics on AI crawlers.
 
-Review the AI crawlers detected on your site in the **Metrics** tab of the Cloudflare dashboard for key metrics.
+</TabItem><TabItem label="Paid plan">
+
+Review the AI crawlers detected on your site.
+
+<Steps>
+1. Go to **AI Audit**.
+2. From the **Metrics** tab, view key metrics on AI crawlers.
+</Steps>
+</TabItem>
+</Tabs>
 
-Refer to [Detect AI crawlers](/ai-audit/features/detect-ai-crawlers/) for more information.
+Refer to [Analyze AI crawlers](/ai-audit/features/analyze-ai-crawlers/) for more information on AI Audit metrics.
 
 ## Related resources
 

src/content/docs/ai-audit/index.mdx

@@ -33,19 +33,19 @@ AI Audit is currently only available as a beta product.
 ## Features
 
 <Feature
-	header="AI crawler detection"
-	href="/ai-audit/features/detect-ai-crawlers/"
+	header="Manage AI crawlers"
+	href="/ai-audit/features/manage-ai-crawlers/"
 	cta="View AI crawlers"
 >
-	Displays information about AI crawlers in your domains' pages.
+	Allow or block individual AI crawlers.
 </Feature>
 
 <Feature
-	header="Enforce robots.txt"
-	href="/ai-audit/features/enforce-robots-txt/"
-	cta="Enforce your robots.txt"
+	header="Analyze AI crawlers"
+	href="/ai-audit/features/analyze-ai-crawlers/"
+	cta="Analyze AI crawler metrics"
 >
-	Enforce your `robots.txt` with a Cloudflare WAF rule.
+	Analyze how AI crawlers interact with your domains.
 </Feature>
 
 ---

src/content/docs/ai-audit/reference/glossary.mdx

@@ -0,0 +1,13 @@
+---
+title: Glossary
+pcx_content_type: glossary
+sidebar:
+  order: 12
+
+---
+
+import { Glossary } from "~/components"
+
+Review the definitions for terms used across Cloudflare's AI Audit documentation.
+
+<Glossary product="ai-audit" />