cloudflare · Oxyjun · Jul 1, 2025 · Jul 1, 2025 · Jul 1, 2025 · Jul 1, 2025
@@ -158,6 +158,9 @@
 /ai-gateway/configuration/websockets-api/non-realtime-api/ /ai-gateway/websockets-api/non-realtime-api/ 301
 /ai-gateway/configuration/websockets-api/realtime-api/ /ai-gateway/websockets-api/realtime-api/ 301
 
+# ai audit
+/ai-audit/features/detect-ai-crawlers/ /ai-audit/features/analyze-ai-crawlers/ 301
+/ai-audit/features/enforce-robots-txt/ /ai-audit/features/analyze-ai-crawlers/ 301
 
 # agents
 /agents/capabilities/mcp-server/ /agents/model-context-protocol/ 301

@@ -0,0 +1,20 @@
+---
+title: Introducing pay per crawl private beta
+description: A new feature which allows site owners to charge crawlers for access to their content
+date: 2025-07-01T12:00:00Z
+---
+
+We are introducing a new feature of AI Audit - pay per crawl. Pay per crawl enables site owners to require payment from AI crawlers every time the crawlers access their content, thereby fostering a fairer Internet by enabling site owners to control and monetize how their content gets used by AI.
+
+![Pay per crawl](~/assets/images/changelog/ai-audit/pay-per-crawl.png)
+
+**For Site Owners:**
+- Set pricing and select which crawlers to charge for content access
+- Manage payments via Stripe
+- Monitor analytics on successful content deliveries
+
+**For AI Crawler Owners:**
+- Use HTTP headers to request and accept pricing
+- Receive clear confirmations on charges for accessed content
+
+Learn more in the [pay per crawl documentation](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/).
@@ -0,0 +1,38 @@
+---
+title: AI Audit with Cloudflare Bots
+pcx_content_type: concept
+sidebar:
+  order: 5
+---
+
+import { GlossaryTooltip, Example } from "~/components";
+
+AI Audit works alongside other Cloudflare products, such as Cloudflare [bot solutions](/bots/). Bot solutions identifies traffic matching patterns of known bots, and can challenge or block the bots as you wish.
+
+## Order of precedence
+
+- AI Audit's AI crawler blocking uses [WAF custom rules](/waf/custom-rules/), which take place before Cloudflare bot solutions.
+- AI Audit's pay per crawl takes place after Cloudflare bot solutions.
+
+```mermaid
+graph LR
+A[Traffic] --> B[WAF custom rules<br>AI Audit: Crawler blocks]
+B --> C[Cloudflare<br>Bot Solutions]
+C --> D[AI Audit:<br>Pay Per Crawl]
+classDef highlight fill:#F6821F,color:white
+```
+
+For more information on how Cloudflare bot solutions works with WAF custom rules, refer to [How it works](/bots/concepts/bot/#how-it-works).
+
+## Examples
+
+Consider the following examples.
+
+### Bot rule which blocks all AI bots vs pay per crawl
+
+You may have both of the following enabled:
+
+- A selection of AI crawlers to be charged through AI Audit's pay per crawl
+- Bot configuration option to [Block AI Bots](/bots/get-started/bot-fight-mode/#block-ai-bots).
+
+Since pay per crawl happens after bot solutions, you need to first turn off **Block AI Bots** to ensure pay per crawl works as intended.
@@ -0,0 +1,81 @@
+---
+title: AI Audit with Cloudflare WAF
+pcx_content_type: concept
+sidebar:
+  order: 2
+
+---
+
+import { GlossaryTooltip, Example, Steps } from "~/components";
+
+AI Audit works alongside other Cloudflare products, such as Cloudflare [Web Application Firewall (WAF)](/waf/). WAF checks incoming web and API requests, and filters undesired traffic based on rules. [WAF custom rules](/waf/custom-rules/) allow you to perform certain actions such as enforcing <GlossaryTooltip term="robots.txt">`robots.txt`</GlossaryTooltip>.
+
+## Order of precedence
+
+- AI Audit uses WAF custom rules to block the selection of AI crawlers the site owner has decided to block.
+- AI Audit's pay per crawl feature takes place after WAF.
+
+```mermaid
+graph LR
+A[Traffic] --> B[WAF custom rules<br>AI Audit: Crawler blocks]
+B --> C[Cloudflare<br>Bot Solutions]
+C --> D[AI Audit:<br>Pay Per Crawl]
+classDef highlight fill:#F6821F,color:white
+```
+
+For this reason, if you plan on using AI Audit to manage AI crawlers, you may wish to modify your existing WAF custom rules such that it does not affect AI crawlers. This will allow you to manage AI crawlers only from AI Audit, thereby streamlining your workflow.
+
+:::note[How AI Audit uses WAF custom rules]
+When you block AI crawlers via AI Audit (either all or some), you are using **one** WAF custom rule to block those AI crawlers.
+
+If you choose to allow all AI crawlers, AI Audit does not utilize any WAF custom rules.
+
+Depending on the type of account you have, you may have a limited number of WAF custom rules.
+:::
+
+## Examples of using WAF vs AI Audit
+
+Consider the following examples.
+
+### Traffic from a restricted country vs pay per crawl
+
+You may have both of the following features enabled:
+
+- [WAF custom rule to block traffic from specific countries](/waf/custom-rules/use-cases/block-traffic-from-specific-countries/)
+- AI Audit's [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) to charge AI crawlers when they request access to your content
+
+Since WAF custom rules are enforced before pay per crawl, traffic (including AI crawlers) from your blocked countries will continue to be blocked, even if they provide the [required headers](/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/#1-include-required-headers) for pay per crawl.
+
+### Allowed search engine bots via WAF custom rule vs pay per crawl
+
+You may have both of the following features enabled:
+
+- [WAF custom rule to allow search engine bots](/waf/custom-rules/use-cases/allow-traffic-from-verified-bots/)
+- AI Audit's [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) to charge all AI crawlers when they request access to your content (including search engine bots).
+
+Since WAF custom rules are enforced before pay per crawl:
+
+- Only search engine bots will be able to access your site (enforced by WAF custom rule).
+- The search engine bots will then be charged for access to your content (enforced by AI Audit's pay per crawl).
+
+:::note
+This example only serves to highlight the order of precedence between WAF and AI Audit.
+
+Practically, it may be beneficial to allow well-behaved search engine bots to access your content to ensure your content is indexed.
+:::
+
+### Conflict in AI crawler blocking logic
+
+You may have both of the following features enabled:
+
+- A WAF custom rule which blocks all bots.
+- AI Audit selection which allows certain AI crawlers.
+
+In this scenario, you have two WAF custom rules, each directing a different logic for handling AI crawlers. To resolve this issue:
+
+<Steps>
+1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
+2. Go to **Rules** > **Overview**.
+3. Identify your WAF custom rule and the AI Audit rule.
+4. Drag the rule you wish to prioritize to the top, or modify your WAF custom rule to ensure it does not conflict with your AI Audit configurations.
+</Steps>
@@ -0,0 +1,12 @@
+---
+title: Configuration
+pcx_content_type: navigation
+sidebar:
+  group:
+    hideIndex: true
+  order: 8
+---
+
+import { DirectoryListing } from "~/components";
+
+<DirectoryListing />
@@ -2,14 +2,14 @@
 title: Analyze AI crawlers
 pcx_content_type: concept
 sidebar:
-  order: 7
+  order: 5
 ---
 
-import { Tabs, TabItem } from "~/components";
+import { Steps, Tabs, TabItem } from "~/components";
 
-AI Audit metrics provides you with key insights on how AI crawlers are interacting with your website.
+AI Audit metrics provide you with insight on how AI crawlers are interacting with your website ([Cloudflare zone](/fundamentals/concepts/accounts-and-zones/#zones)).
 
-To analyze AI crawlers:
+To view AI Audit metrics:
 
 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
 2. Go to **AI Audit**.
@@ -19,23 +19,29 @@ To analyze AI crawlers:
 
 AI Audit provides you with the following metrics to help you understand how AI crawlers are interacting with your website.
 
-| Metric                             | Description                                                              |
-| ---------------------------------- | ------------------------------------------------------------------------ |
-| Total requests                     | The total number of requests to crawl your website, from all AI crawlers |
-| Allowed requests                   | The number of requests you have allowed (by allowing AI crawlers)        |
-| Blocked requests                   | The number of requests you have blocked (by blocking AI crawlers)        |
-| Requests by AI crawlers            | A graph which displays the number of crawl requests from each AI crawler |
-| Most popular paths by AI crawlers  | The most popular pages crawled by AI crawlers, for each AI crawler       |
-| Hosts                              | Top five most requested hosts (website domains)                          |
-| Paths                              | Top five most requested paths                                            |
-| AI crawler operators               | Top five most active AI crawler operators (by requests)                  |
+| Metric                                | Description                                                                   |
+| ------------------------------------- | ----------------------------------------------------------------------------- |
+| Total requests                        | The total number of requests to crawl your website, from all AI crawlers      |
+| Blocked requests                      | The number of requests you have blocked (by blocking AI crawlers)             |
+| Allowed requests                      | The number of requests you have allowed (by allowing AI crawlers)             |
+| Request by AI crawlers                | A graph which displays the number of crawl requests from each AI crawler      |
+| Most popular paths by AI crawlers     | The most popular pages crawled by AI crawlers, for each AI crawler            |
+| Hosts                                 | The owner of the AI crawler                                                   |
+| Overall popular paths                 | The most popular pages crawled by AI crawlers, from all AI crawlers           |
+| Most active AI crawlers by operators  | The AI crawler owners with the highest number of requests to access your site |
+
+:::note[Requests in AI Audit metrics]
+The number of requests in AI Audit metrics are specifically requests which were met with HTTP code 200 (the request was successfully served, with actual content).
+
+AI Audit metrics filter all other HTTP codes.
+:::
 
 ## Filter date range
 
 You can use the date filter to choose the period of time you wish to analyze.
 
 <Tabs>
-<TabItem label="Free plan">
+<TabItem label="Free plans">
 
 Filter options:
 
@@ -53,4 +59,4 @@ Filter options:
 </TabItem>
 </Tabs>
 
-The values of the AI Audit metrics will update according to your filter.
+The values of the AI Audit metrics will update according to your filter.
@@ -5,7 +5,7 @@ sidebar:
   order: 2
 ---
 
-import { Steps, GlossaryTooltip, Tabs, TabItem } from "~/components";
+import { Steps, GlossaryTooltip, Tabs, TabItem, Example } from "~/components";
 
 AI Audit enables you to take specific action for each AI crawler.
 
@@ -17,9 +17,9 @@ To manage AI crawlers:
 
 ## View the list of AI crawlers
 
-The **AI Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information, depending on the type of plan you are on.
+The **AI Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information.
 
-<Tabs syncKey ="plan">
+<Tabs syncKey="plan">
 <TabItem label="Free plan">
 | Column                | Details                                                                                                                                                             |
 | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -29,24 +29,75 @@ The **AI Crawlers** tab displays a table of AI crawlers that are requesting acce
 | Block                 | Toggle for blocking specific AI crawlers.                                                                                                                           |
 
 </TabItem> <TabItem label="Paid plan">
-
 | Column                | Details                                                                                                                                                             |
 | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | AI Crawlers           | The name of the AI crawler.                                                                                                                                         |
 | Operator              | The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers.                                                                |
 | Category              | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/).                                                   |
 | Requests              | The number of times the AI crawler has requested to crawl your content.                                                                                             |
 | Robots.txt violations | The number of times the AI crawler has violated your <GlossaryTooltip term="robots.txt">`robots.txt`</GlossaryTooltip> file.                                        |
-| Block                 | Toggle for blocking specific AI crawlers.                                                                                                                           |
+| Action                | The action you wish to take for the AI crawler. Refer to [Take action for each AI crawler](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). |
 
 </TabItem>
 </Tabs>
 
-## Block access
+## Take action for each AI crawler
+
+<Tabs>
+<TabItem label="With pay per crawl">
+
+For each AI crawler, you can take one of three actions: allow, charge, or block.
+
+<Example title="Allow access">
+
+You can allow an AI crawler to scrape your content.
+
+From the **Actions** column, select **Allow**.
+
+Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
+
+For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
+
+</Example>
+
+<Example title="Charge for crawl (Enterprise only)">
+
+You can charge the owner of the AI crawler for each crawl request.
+
+From the **Actions** column, select **Charge**.
+
+For more information, refer to [What is Pay Per Crawl?](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/).
+
+</Example>
+
+<Example title="Block access">
 
 You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
 
+From the **Actions** column, select **Block**.
+</Example>
+
+</TabItem>
+<TabItem label="Without pay per crawl">
+
+For each AI crawler, you can choose to allow or block access.
+
+<Example title="Allow access">
+
+You can allow an AI crawler to scrape your content. Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
+
+For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
+
+</Example>
+
+<Example title="Block access">
+You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
+
 Use the toggles in the **Block** column to block specific AI crawlers from accessing your content.
+</Example>
+
+</TabItem>
+</Tabs>
 
 ## Filter AI crawler data
 
@@ -57,3 +108,7 @@ You can use filters to narrow the scope of your result.
 - **Category:** Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).
 
 The values of the table will update according to your filter.
+
+## Related resources
+
+- Use [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) to charge AI crawlers every time they access your content.
@@ -0,0 +1,12 @@
+---
+title: Pay Per Crawl
+pcx_content_type: navigation
+sidebar:
+  group:
+    hideIndex: true
+  order: 20
+---
+
+import { DirectoryListing } from "~/components";
+
+<DirectoryListing />
@@ -0,0 +1,43 @@
+---
+title: Connect Stripe
+pcx_content_type: concept
+sidebar:
+  order: 4
+---
+
+import { Steps } from "~/components";
+
+```mermaid
+graph LR
+A[Set up your<br>Cloudflare Account] --> B[Connect to<br>Stripe]:::highlight
+B --> C[Verify your<br>AI crawler]
+C --> D[Crawl pages]
+classDef highlight fill:#F6821F,color:white
+```
+
+Connect your Cloudflare account to Stripe to process payments. Pay per crawl uses Stripe to process payments between AI crawler owners and site owners.
+
+{/* prettier-ignore */}
+<Steps>
+1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account.
+2. Go to **Manage Account** > **Configurations**.
+3. Go to the **Pay Per Crawl** tab.
+4. From **Connect to Stripe**, select **Connect**.
+5. Select **Continue to Stripe**.
+6. Follow the on-screen instructions to connect your Cloudflare account to a Stripe account.
+   :::note[Use a non-personal email address]
+   Cloudflare recommends using a dedicated email to manage your pay per crawl account (for example, `[email protected]`).
+   :::
+</Steps>
+
+When you successfully connect Stripe to your account, you will see a green tick ✅ next to **Stripe connection**.
+
+:::caution[Spending limits]
+Cloudflare is not responsible for configuring spending limits. Ensure you have configured a maximum spending limit on your AI crawler.
+:::
+
+## Billing
+
+Charges are recorded upon successful delivery of content that is requested with valid [crawler price headers](/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/#22-include-payment-headers).
+
+Invoices are created and managed via Stripe. Crawlers are responsible for setting and enforcing their own spending limits.