[AI Audit] AI Audit update docs

public/__redirects

@@ -138,6 +138,10 @@
 # ai
 /ai/ /use-cases/ai/ 301
 
+# ai-audit
+/ai-audit/features/detect-ai-crawlers/ /ai-audit/features/analyze-ai-crawlers/ 301
+/ai-audit/features/enforce-robots-txt/ /ai-audit/features/analyze-ai-crawlers/ 301
+
 # AI Gateway
 /ai-gateway/get-started/configuring-settings/ /ai-gateway/get-started/ 301
 /ai-gateway/get-started/connecting-applications/ /ai-gateway/get-started/ 301

src/content/changelog/ai-audit/2024-09-23-ai-audit-launch.mdx

@@ -9,7 +9,7 @@ Every site on Cloudflare now has access to [**AI Audit**](/ai-audit/), which sum
 You can use this data to:
 
 - Understand how and how often crawlers access your site (and which content is the most popular).
-- Block some or all of the AI bots accessing your site.
+- Block specific AI bots accessing your site.
 - Use Cloudflare to enforce your `robots.txt` policy via an automatic WAF rule.
 
 ![View AI bot activity with AI Audit](~/assets/images/changelog/ai-audit/ai-audit-overview.png)

src/content/changelog/ai-audit/2025-06-30-refresh.mdx

@@ -0,0 +1,19 @@
+---
+title: AI Audit refresh
+description: More intuitive AI crawler management
+date: 2025-07-01T11:00:00Z
+---
+
+We redesigned the AI Audit dashboard to provide more intuitive and granular control over AI crawlers.
+
+- From the new **AI Crawlers** tab: block specific AI crawlers.
+- From the new **Metrics** tab: view AI Audit metrics.
+
+![Block AI crawlers](~/assets/images/changelog/ai-audit/manage-ai-crawlers.png)
+
+![Analyze AI crawler activity](~/assets/images/changelog/ai-audit/analyze-metrics.png)
+
+To get started, explore:
+
+- [Manage AI crawlers](/ai-audit/features/manage-ai-crawlers/).
+- [Analyze AI crawlers](/ai-audit/features/analyze-ai-crawlers/).

src/content/docs/ai-audit/features/analyze-ai-crawlers.mdx

@@ -0,0 +1,56 @@
+---
+title: Analyze AI crawlers
+pcx_content_type: concept
+sidebar:
+  order: 7
+---
+
+import { Tabs, TabItem } from "~/components";
+
+AI Audit metrics provides you with key insights on how AI crawlers are interacting with your website.
+
+To analyze AI crawlers:
+
+1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
+2. Go to **AI Audit**.
+3. Go to the **Metrics** tab.
+
+## View AI Audit metrics
+
+AI Audit provides you with the following metrics to help you understand how AI crawlers are interacting with your website.
+
+| Metric                             | Description                                                              |
+| ---------------------------------- | ------------------------------------------------------------------------ |
+| Total requests                     | The total number of requests to crawl your website, from all AI crawlers |
+| Allowed requests                   | The number of requests you have allowed (by allowing AI crawlers)        |
+| Blocked requests                   | The number of requests you have blocked (by blocking AI crawlers)        |
+| Requests by AI crawlers            | A graph which displays the number of crawl requests from each AI crawler |
+| Most popular paths by AI crawlers  | The most popular pages crawled by AI crawlers, for each AI crawler       |
+| Hosts                              | Top five most requested hosts (website domains)                          |
+| Paths                              | Top five most requested paths                                            |
+| AI crawler operators               | Top five most active AI crawler operators (by requests)                  |
+
+## Filter date range
+
+You can use the date filter to choose the period of time you wish to analyze.
+
+<Tabs>
+<TabItem label="Free plan">
+
+Filter options:
+
+- Past 24 hours
+
+</TabItem><TabItem label="Paid plans">
+
+Filter options:
+
+- Past 24 hours
+- Past 7 days
+- Past 14 days
+- Past month
+
+</TabItem>
+</Tabs>
+
+The values of the AI Audit metrics will update according to your filter.

src/content/docs/ai-audit/features/manage-ai-crawlers.mdx

@@ -0,0 +1,59 @@
+---
+title: Manage AI crawlers
+pcx_content_type: concept
+sidebar:
+  order: 2
+---
+
+import { Steps, GlossaryTooltip, Tabs, TabItem } from "~/components";
+
+AI Audit enables you to take specific action for each AI crawler.
+
+To manage AI crawlers:
+
+1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
+2. Go to **AI Audit**.
+3. Go to the **AI Crawlers** tab.
+
+## View the list of AI crawlers
+
+The **AI Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information, depending on the type of plan you are on.
+
+<Tabs syncKey ="plan">
+<TabItem label="Free plan">
+| Column                | Details                                                                                                                                                             |
+| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| AI Crawlers           | The name of the AI crawler.                                                                                                                                         |
+| Operator              | The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers.                                                                |
+| Category              | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/).                                                   |
+| Block                 | Toggle for blocking specific AI crawlers.                                                                                                                           |
+
+</TabItem> <TabItem label="Paid plan">
+
+| Column                | Details                                                                                                                                                             |
+| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| AI Crawlers           | The name of the AI crawler.                                                                                                                                         |
+| Operator              | The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers.                                                                |
+| Category              | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/).                                                   |
+| Requests              | The number of times the AI crawler has requested to crawl your content.                                                                                             |
+| Robots.txt violations | The number of times the AI crawler has violated your <GlossaryTooltip term="robots.txt">`robots.txt`</GlossaryTooltip> file.                                        |
+| Block                 | Toggle for blocking specific AI crawlers.                                                                                                                           |
+
+</TabItem>
+</Tabs>
+
+## Block access
+
+You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
+
+Use the toggles in the **Block** column to block specific AI crawlers from accessing your content.
+
+## Filter AI crawler data
+
+You can use filters to narrow the scope of your result.
+
+- **Name:** Search the name of the AI crawler.
+- **Operator:** Filter by the AI crawler operator.
+- **Category:** Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).
+
+The values of the table will update according to your filter.

src/content/docs/ai-audit/get-started.mdx

@@ -11,76 +11,77 @@ head:
 description: Learn how to set up AI Audit.
 ---
 
-import { Details, Render, Steps } from "~/components";
+import { Details, Render, Steps, Tabs, TabItem, GlossaryTooltip } from "~/components";
 
 This guide instructs you on how to:
 
 - View AI crawlers that are interacting with pages in your domain (a [Cloudflare zone](/fundamentals/concepts/accounts-and-zones/#zones)).
-- Create a rule to block AI crawlers on your pages.
+- Use AI Audit to block individual crawlers from accessing your content.
 
 ## Prerequisites
 
 1. Sign up for a [Cloudflare account](https://dash.cloudflare.com/sign-up/).
 2. [Connect your domain to Cloudflare](/fundamentals/manage-domains/add-site/).
 3. Make sure your domain is [proxying traffic through Cloudflare](/fundamentals/concepts/how-cloudflare-works/#cloudflare-as-a-reverse-proxy).
 
-## 1. Block all AI crawlers
+## 1. Block specific AI crawlers
 
-To use AI Audit:
+<Tabs syncKey="plans">
+<TabItem label="Free plan">
+
+To only block specific AI crawlers:
 
 {/* prettier-ignore */}
 <Steps>
-1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
-2. Go to **AI Audit**.
-3. From **Most Popular Paths**, select **Block All**.
-4. From the **Bot traffic** page, under **Block AI Bots**, select **Enable**.
+1. Go to **AI Audit**.
+2. From **AI Crawlers** tab, go to the list of AI crawlers.
+3. Review the AI crawlers that are accessing your domain.
+4. From the **Block** column, toggle the switch for AI crawlers you with to block.
 </Steps>
 
-This feature also enables Cloudflare's [Bot Fight Mode: Block AI bots](/bots/get-started/bot-fight-mode/#block-ai-bots).
-
-You can also create more complex rules when taking action on AI crawlers. For more information on creating more specific rules, refer to [Create a custom rule in the dashboard
-](/waf/custom-rules/create-dashboard/).
-
-<Details header="Which bots will Cloudflare block?">
-	<Render file="list-ai-bots" product="bots" />
-</Details>
-
-:::note
-
-For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
+:::note[Quality of AI crawler detection]
+On the free plan, AI Audit identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Audit to detect easy-to-detect (well-known) AI crawlers.
 
+Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field.
 :::
 
-## 2. Block specific bot categories (Enterprise plan only)
+</TabItem> <TabItem label="Paid plans">
 
-Customers on the Enterprise plan -- and with a [Bot Management subscription](/bots/plans/bm-subscription/) -- can choose to only block specific AI crawlers, while allowing others.
+To only block specific AI crawlers:
 
 {/* prettier-ignore */}
 <Steps>
-1. Go to the **AI Audit**.
-2. From **Most Popular Paths**, select **Block Some**.
-3. From the **Security rules** page, select **Create rule** > **Custom rules**.
-4. Provide a name for the custom rule. For example, "Block unwanted AI crawlers".
-5. From the **Field** dropdown, select **Verified Bot Category**.
-6. From the **Value** dropdown, select the specific bot category you wish to block.
-   - You can use **And** / **Or** buttons to add additional conditions. For example, you can use multiple **Or** options to include multiple bot categories in the same rule.
-7. From the **Then take action...** section:
-   - For **Choose action**, select **Block**.
-   - For **With response type**, select **Default Cloudflare WAF block page**.
-8. From the **Place at** section:
-   - For **Select order**, select **First**.
-9. Select **Save**.
+1. Go to **AI Audit**.
+2. From **AI Crawlers** tab, go to the list of AI crawlers.
+3. Review key information such as:
+   - Bot operator
+   - Number of requests sent by the AI crawler
+   - Whether the AI crawler is allowed according to your `robots.txt`, or the number of times the AI crawler has violated your `robots.txt`.
+4. From the **Block** column, toggle the switch for AI crawlers you with to block.
 </Steps>
 
-This custom rule will only block the AI bots which belong to the [verified bot categories](/bots/concepts/bot/verified-bots/categories/) you have included in your rule (in step 6).
+</TabItem>
+</Tabs>
 
-For more information on creating a custom WAF rule, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
+For more information, refer to [Manage AI crawlers](/ai-audit/features/manage-ai-crawlers/).
 
-## 3. Review detected AI crawlers
+You can also create more complex rules when taking action on AI crawlers, using [Cloudflare WAF](/waf/). For more information on creating more specific rules, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/).
 
-Review the AI crawlers detected on your site in the **Metrics** tab of the Cloudflare dashboard for key metrics.
+## 2. Review detected AI crawlers
+
+<Tabs>
+<TabItem label="All plans">
+
+Review the AI crawlers detected on your site.
+
+<Steps>
+1. Go to **AI Audit**.
+2. From the **Metrics** tab, view key metrics on AI crawlers.
+</Steps>
+</TabItem>
+</Tabs>
 
-Refer to [Detect AI crawlers](/ai-audit/features/detect-ai-crawlers/) for more information.
+Note that on free plans, the **Metrics** tab only displays metrics for the past 24 hours.
 
 ## Related resources
 

src/content/docs/ai-audit/index.mdx

@@ -33,19 +33,19 @@ AI Audit is currently only available as a beta product.
 ## Features
 
 <Feature
-	header="AI crawler detection"
-	href="/ai-audit/features/detect-ai-crawlers/"
+	header="Manage AI crawlers"
+	href="/ai-audit/features/manage-ai-crawlers/"
 	cta="View AI crawlers"
 >
-	Displays information about AI crawlers in your domains' pages.
+	Allow or block individual AI crawlers.
 </Feature>
 
 <Feature
-	header="Enforce robots.txt"
-	href="/ai-audit/features/enforce-robots-txt/"
-	cta="Enforce your robots.txt"
+	header="Analyze AI crawlers"
+	href="/ai-audit/features/analyze-ai-crawlers/"
+	cta="Analyze AI crawler metrics"
 >
-	Enforce your `robots.txt` with a Cloudflare WAF rule.
+	Analyze how AI crawlers interact with your domains.
 </Feature>
 
 ---