diff --git a/public/__redirects b/public/__redirects index ce96500f042e428..bfd754f25979be4 100644 --- a/public/__redirects +++ b/public/__redirects @@ -158,7 +158,6 @@ /ai-gateway/configuration/websockets-api/non-realtime-api/ /ai-gateway/websockets-api/non-realtime-api/ 301 /ai-gateway/configuration/websockets-api/realtime-api/ /ai-gateway/websockets-api/realtime-api/ 301 - # agents /agents/capabilities/mcp-server/ /agents/model-context-protocol/ 301 diff --git a/src/assets/images/ai-audit/ai-audit-pay-per-crawl-diagram.png b/src/assets/images/ai-audit/ai-audit-pay-per-crawl-diagram.png new file mode 100644 index 000000000000000..29e95c0335f4220 Binary files /dev/null and b/src/assets/images/ai-audit/ai-audit-pay-per-crawl-diagram.png differ diff --git a/src/assets/images/changelog/ai-audit/pay-per-crawl.png b/src/assets/images/changelog/ai-audit/pay-per-crawl.png new file mode 100644 index 000000000000000..2c36b196b71f170 Binary files /dev/null and b/src/assets/images/changelog/ai-audit/pay-per-crawl.png differ diff --git a/src/content/changelog/ai-audit/2025-07-01-pay-per-crawl.mdx b/src/content/changelog/ai-audit/2025-07-01-pay-per-crawl.mdx new file mode 100644 index 000000000000000..5003f3f80815bdd --- /dev/null +++ b/src/content/changelog/ai-audit/2025-07-01-pay-per-crawl.mdx @@ -0,0 +1,20 @@ +--- +title: Introducing Pay Per Crawl (private beta) +description: A new feature which allows site owners to charge crawlers for access to their content +date: 2025-07-01T12:00:00Z +--- + +We are introducing a new feature of [AI Audit](/ai-audit/) — Pay Per Crawl. [Pay Per Crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) enables site owners to require payment from AI crawlers every time the crawlers access their content, thereby fostering a fairer Internet by enabling site owners to control and monetize how their content gets used by AI. + +![Pay per crawl](~/assets/images/changelog/ai-audit/pay-per-crawl.png) + +**For Site Owners:** +- Set pricing and select which crawlers to charge for content access +- Manage payments via Stripe +- Monitor analytics on successful content deliveries + +**For AI Crawler Owners:** +- Use HTTP headers to request and accept pricing +- Receive clear confirmations on charges for accessed content + +Learn more in the [Pay Per Crawl documentation](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/). diff --git a/src/content/changelog/ai-audit/2025-06-30-refresh.mdx b/src/content/changelog/ai-audit/2025-07-01-refresh.mdx similarity index 100% rename from src/content/changelog/ai-audit/2025-06-30-refresh.mdx rename to src/content/changelog/ai-audit/2025-07-01-refresh.mdx diff --git a/src/content/docs/ai-audit/configuration/ai-audit-with-bots.mdx b/src/content/docs/ai-audit/configuration/ai-audit-with-bots.mdx new file mode 100644 index 000000000000000..cea31e83922993d --- /dev/null +++ b/src/content/docs/ai-audit/configuration/ai-audit-with-bots.mdx @@ -0,0 +1,38 @@ +--- +title: AI Audit with Cloudflare Bots +pcx_content_type: concept +sidebar: + order: 5 +--- + +import { GlossaryTooltip, Example } from "~/components"; + +AI Audit works alongside other Cloudflare products, such as Cloudflare [bot solutions](/bots/). Bot solutions identifies traffic matching patterns of known bots, and can challenge or block the bots as you wish. + +## Order of precedence + +- AI Audit's AI crawler blocking uses [WAF custom rules](/waf/custom-rules/), which take place before Cloudflare bot solutions. +- AI Audit's pay per crawl takes place after Cloudflare bot solutions. + +```mermaid +graph LR +A[Traffic] --> B[WAF custom rules
AI Audit: Crawler blocks] +B --> C[Cloudflare
Bot Solutions] +C --> D[AI Audit:
Pay Per Crawl] +classDef highlight fill:#F6821F,color:white +``` + +For more information on how Cloudflare bot solutions works with WAF custom rules, refer to [How it works](/bots/concepts/bot/#how-it-works). + +## Examples + +Consider the following examples. + +### Bot rule which blocks all AI bots vs pay per crawl + +You may have both of the following enabled: + +- A selection of AI crawlers to be charged through AI Audit's pay per crawl +- Bot configuration option to [Block AI Bots](/bots/get-started/bot-fight-mode/#block-ai-bots). + +Since pay per crawl happens after bot solutions, you need to first turn off **Block AI Bots** to ensure pay per crawl works as intended. \ No newline at end of file diff --git a/src/content/docs/ai-audit/configuration/ai-audit-with-waf.mdx b/src/content/docs/ai-audit/configuration/ai-audit-with-waf.mdx new file mode 100644 index 000000000000000..133273882b52000 --- /dev/null +++ b/src/content/docs/ai-audit/configuration/ai-audit-with-waf.mdx @@ -0,0 +1,81 @@ +--- +title: AI Audit with Cloudflare WAF +pcx_content_type: concept +sidebar: + order: 2 + +--- + +import { GlossaryTooltip, Example, Steps } from "~/components"; + +AI Audit works alongside other Cloudflare products, such as Cloudflare [Web Application Firewall (WAF)](/waf/). WAF checks incoming web and API requests, and filters undesired traffic based on rules. [WAF custom rules](/waf/custom-rules/) allow you to perform certain actions such as enforcing `robots.txt`. + +## Order of precedence + +- AI Audit uses WAF custom rules to block the selection of AI crawlers the site owner has decided to block. +- AI Audit's pay per crawl feature takes place after WAF. + +```mermaid +graph LR +A[Traffic] --> B[WAF custom rules
AI Audit: Crawler blocks] +B --> C[Cloudflare
Bot Solutions] +C --> D[AI Audit:
Pay Per Crawl] +classDef highlight fill:#F6821F,color:white +``` + +For this reason, if you plan on using AI Audit to manage AI crawlers, you may wish to modify your existing WAF custom rules such that it does not affect AI crawlers. This will allow you to manage AI crawlers only from AI Audit, thereby streamlining your workflow. + +:::note[How AI Audit uses WAF custom rules] +When you block AI crawlers via AI Audit (either all or some), you are using **one** WAF custom rule to block those AI crawlers. + +If you choose to allow all AI crawlers, AI Audit does not utilize any WAF custom rules. + +Depending on the type of account you have, you may have a limited number of WAF custom rules. +::: + +## Examples of using WAF vs AI Audit + +Consider the following examples. + +### Traffic from a restricted country vs pay per crawl + +You may have both of the following features enabled: + +- [WAF custom rule to block traffic from specific countries](/waf/custom-rules/use-cases/block-traffic-from-specific-countries/) +- AI Audit's [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) to charge AI crawlers when they request access to your content + +Since WAF custom rules are enforced before pay per crawl, traffic (including AI crawlers) from your blocked countries will continue to be blocked, even if they provide the [required headers](/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/#1-include-required-headers) for pay per crawl. + +### Allowed search engine bots via WAF custom rule vs pay per crawl + +You may have both of the following features enabled: + +- [WAF custom rule to allow search engine bots](/waf/custom-rules/use-cases/allow-traffic-from-verified-bots/) +- AI Audit's [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) to charge all AI crawlers when they request access to your content (including search engine bots). + +Since WAF custom rules are enforced before pay per crawl: + +- Only search engine bots will be able to access your site (enforced by WAF custom rule). +- The search engine bots will then be charged for access to your content (enforced by AI Audit's pay per crawl). + +:::note +This example only serves to highlight the order of precedence between WAF and AI Audit. + +Practically, it may be beneficial to allow well-behaved search engine bots to access your content to ensure your content is indexed. +::: + +### Conflict in AI crawler blocking logic + +You may have both of the following features enabled: + +- A WAF custom rule which blocks all bots. +- AI Audit selection which allows certain AI crawlers. + +In this scenario, you have two WAF custom rules, each directing a different logic for handling AI crawlers. To resolve this issue: + + +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain. +2. Go to **Rules** > **Overview**. +3. Identify your WAF custom rule and the AI Audit rule. +4. Drag the rule you wish to prioritize to the top, or modify your WAF custom rule to ensure it does not conflict with your AI Audit configurations. + \ No newline at end of file diff --git a/src/content/docs/ai-audit/configuration/index.mdx b/src/content/docs/ai-audit/configuration/index.mdx new file mode 100644 index 000000000000000..e77a2bf0d6dc039 --- /dev/null +++ b/src/content/docs/ai-audit/configuration/index.mdx @@ -0,0 +1,12 @@ +--- +title: Configuration +pcx_content_type: navigation +sidebar: + group: + hideIndex: true + order: 8 +--- + +import { DirectoryListing } from "~/components"; + + diff --git a/src/content/docs/ai-audit/features/analyze-ai-crawlers.mdx b/src/content/docs/ai-audit/features/analyze-ai-crawlers.mdx index 7a6aa6fec86cc6e..7ba90e14f5976da 100644 --- a/src/content/docs/ai-audit/features/analyze-ai-crawlers.mdx +++ b/src/content/docs/ai-audit/features/analyze-ai-crawlers.mdx @@ -2,14 +2,14 @@ title: Analyze AI crawlers pcx_content_type: concept sidebar: - order: 7 + order: 5 --- -import { Tabs, TabItem } from "~/components"; +import { Steps, Tabs, TabItem } from "~/components"; -AI Audit metrics provides you with key insights on how AI crawlers are interacting with your website. +AI Audit metrics provide you with insight on how AI crawlers are interacting with your website ([Cloudflare zone](/fundamentals/concepts/accounts-and-zones/#zones)). -To analyze AI crawlers: +To view AI Audit metrics: 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain. 2. Go to **AI Audit**. @@ -19,23 +19,29 @@ To analyze AI crawlers: AI Audit provides you with the following metrics to help you understand how AI crawlers are interacting with your website. -| Metric | Description | -| ---------------------------------- | ------------------------------------------------------------------------ | -| Total requests | The total number of requests to crawl your website, from all AI crawlers | -| Allowed requests | The number of requests you have allowed (by allowing AI crawlers) | -| Blocked requests | The number of requests you have blocked (by blocking AI crawlers) | -| Requests by AI crawlers | A graph which displays the number of crawl requests from each AI crawler | -| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler | -| Hosts | Top five most requested hosts (website domains) | -| Paths | Top five most requested paths | -| AI crawler operators | Top five most active AI crawler operators (by requests) | +| Metric | Description | +| ------------------------------------- | ----------------------------------------------------------------------------- | +| Total requests | The total number of requests to crawl your website, from all AI crawlers | +| Blocked requests | The number of requests you have blocked (by blocking AI crawlers) | +| Allowed requests | The number of requests you have allowed (by allowing AI crawlers) | +| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler | +| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler | +| Hosts | The owner of the AI crawler | +| Overall popular paths | The most popular pages crawled by AI crawlers, from all AI crawlers | +| Most active AI crawlers by operators | The AI crawler owners with the highest number of requests to access your site | + +:::note[Requests in AI Audit metrics] +The number of requests in AI Audit metrics are specifically requests which were met with HTTP code 200 (the request was successfully served, with actual content). + +AI Audit metrics filter all other HTTP codes. +::: ## Filter date range You can use the date filter to choose the period of time you wish to analyze. - + Filter options: @@ -53,4 +59,4 @@ Filter options: -The values of the AI Audit metrics will update according to your filter. \ No newline at end of file +The values of the AI Audit metrics will update according to your filter. diff --git a/src/content/docs/ai-audit/features/manage-ai-crawlers.mdx b/src/content/docs/ai-audit/features/manage-ai-crawlers.mdx index 951a9289b2fc94a..0f94f1897d3d195 100644 --- a/src/content/docs/ai-audit/features/manage-ai-crawlers.mdx +++ b/src/content/docs/ai-audit/features/manage-ai-crawlers.mdx @@ -5,7 +5,7 @@ sidebar: order: 2 --- -import { Steps, GlossaryTooltip, Tabs, TabItem } from "~/components"; +import { Steps, GlossaryTooltip, Tabs, TabItem, Example } from "~/components"; AI Audit enables you to take specific action for each AI crawler. @@ -17,9 +17,9 @@ To manage AI crawlers: ## View the list of AI crawlers -The **AI Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information, depending on the type of plan you are on. +The **AI Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information. - + | Column | Details | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -29,7 +29,6 @@ The **AI Crawlers** tab displays a table of AI crawlers that are requesting acce | Block | Toggle for blocking specific AI crawlers. | - | Column | Details | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AI Crawlers | The name of the AI crawler. | @@ -37,16 +36,68 @@ The **AI Crawlers** tab displays a table of AI crawlers that are requesting acce | Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/categories/). | | Requests | The number of times the AI crawler has requested to crawl your content. | | Robots.txt violations | The number of times the AI crawler has violated your `robots.txt` file. | -| Block | Toggle for blocking specific AI crawlers. | +| Action | The action you wish to take for the AI crawler. Refer to [Take action for each AI crawler](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). | -## Block access +## Take action for each AI crawler + + + + +For each AI crawler, you can take one of three actions: allow, charge, or block. + + + +You can allow an AI crawler to scrape your content. + +From the **Actions** column, select **Allow**. + +Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). + +For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works). + + + + + +You can charge the owner of the AI crawler for each crawl request. + +From the **Actions** column, select **Charge**. + +For more information, refer to [What is Pay Per Crawl?](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/). + + + + You can block an AI crawler to completely stop the AI crawler from scraping your webpage. +From the **Actions** column, select **Block**. + + + + + +For each AI crawler, you can choose to allow or block access. + + + +You can allow an AI crawler to scrape your content. Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). + +For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works). + + + + +You can block an AI crawler to completely stop the AI crawler from scraping your webpage. + Use the toggles in the **Block** column to block specific AI crawlers from accessing your content. + + + + ## Filter AI crawler data @@ -57,3 +108,7 @@ You can use filters to narrow the scope of your result. - **Category:** Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver). The values of the table will update according to your filter. + +## Related resources + +- Use [pay per crawl](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) to charge AI crawlers every time they access your content. diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/index.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/index.mdx new file mode 100644 index 000000000000000..81d4a3b313acdfa --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/index.mdx @@ -0,0 +1,12 @@ +--- +title: Pay Per Crawl +pcx_content_type: navigation +sidebar: + group: + hideIndex: true + order: 20 +--- + +import { DirectoryListing } from "~/components"; + + diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/connect-to-stripe.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/connect-to-stripe.mdx new file mode 100644 index 000000000000000..22f2998b0e490fc --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/connect-to-stripe.mdx @@ -0,0 +1,43 @@ +--- +title: Connect Stripe +pcx_content_type: concept +sidebar: + order: 4 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Set up your
Cloudflare Account] --> B[Connect to
Stripe]:::highlight +B --> C[Verify your
AI crawler] +C --> D[Crawl pages] +classDef highlight fill:#F6821F,color:white +``` + +Connect your Cloudflare account to Stripe to process payments. Pay per crawl uses Stripe to process payments between AI crawler owners and site owners. + +{/* prettier-ignore */} + +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account. +2. Go to **Manage Account** > **Configurations**. +3. Go to the **Pay Per Crawl** tab. +4. From **Connect to Stripe**, select **Connect**. +5. Select **Continue to Stripe**. +6. Follow the on-screen instructions to connect your Cloudflare account to a Stripe account. + :::note[Use a non-personal email address] + Cloudflare recommends using a dedicated email to manage your pay per crawl account (for example, `aicrawler@company.com`). + ::: + + +When you successfully connect Stripe to your account, you will see a green tick ✅ next to **Stripe connection**. + +:::caution[Spending limits] +Cloudflare is not responsible for configuring spending limits. Ensure you have configured a maximum spending limit on your AI crawler. +::: + +## Billing + +Charges are recorded upon successful delivery of content that is requested with valid [crawler price headers](/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/#22-include-payment-headers). + +Invoices are created and managed via Stripe. Crawlers are responsible for setting and enforcing their own spending limits. \ No newline at end of file diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx new file mode 100644 index 000000000000000..ef4f3729857c258 --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx @@ -0,0 +1,88 @@ +--- +title: Crawl pages +pcx_content_type: concept +sidebar: + order: 15 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Set up your
Cloudflare Account] --> B[Connect to
Stripe] +B --> C[Verify your
AI crawler] +C --> D[Crawl pages]:::highlight +classDef highlight fill:#F6821F,color:white +``` + +Once your AI crawler complies with Web Bot Auth, you can begin to crawl webpages. For more information on how pay per crawl works, refer to [What is Pay Per Crawl?](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/) + +## 1. Identify payment requirements + +When an AI crawler makes a request to a page protected by pay per crawl, the server will respond with `HTTP/2 402 Payment Required`. This response will also include the `crawler-price` header which specifies the cost to access the content. For example, a response may look like the following: + +```txt +HTTP/2 200 +date: Fri, 06 Jun 2025 08:42:38 GMT +crawler-charged: USD 0.01 +``` + +To access this content, the AI crawler must provide headers for paid access. + +## 2. Access paid content + +### 2.1. Include WBA headers + +Include Web Bot Auth headers by following the steps [Sign your requests](/bots/concepts/bot/verified-bots/web-bot-auth/#4-after-verification-sign-your-requests) + +### 2.2. Include payment headers + +Your AI crawler can specify the price it is willing to pay by providing one of two headers: + +- `crawler-exact-price`: This is the exact price the AI crawler is configured to pay for access. This value should exactly match the price set in the response header `crawler-price`. Include this header in your second request if your AI crawler first received a HTTP status code 402, and you wish to access the content by paying the `crawler-price`. +- `crawler-max-price`: This is the maximum price the AI crawler is configured to pay for access on any content. Use this option if you wish to access any pay per crawl content with a crawl price equal or lower than the maximum price. If a page's `crawler-price` is higher than your `crawler-max-price`, your AI crawler will receive a HTTP 402 response. + +:::note +Note that if a content owner has explicitly blocked your AI crawler (instead of charging your crawler), you cannot access their content, even if you provide `crawler-exact-price` or `crawler-max-price` headers. +::: + +### 2.3. Review response headers + +When you specify a header to indicate payment, you may receive one of two responses: + +#### Successful HTTP/2 200 response + +The value of the `crawler-charged` header indicates the exact amount that will be billed to your Cloudflare account for the request. + +```txt +HTTP/2 200 +date: Fri, 06 Jun 2025 08:42:38 GMT +crawler-charged: USD 0.01 +``` + +#### Unsuccessful HTTP/2 402 response + +If the request is unsuccessful due to an incorrect price header, you will receive a response with the `crawler-price` header. Your request header values must be adjusted to match the `crawler-price`. + +```txt +HTTP/2 402 +date: Fri, 06 Jun 2025 08:42:38 GMT +content-type: text/plain; charset=utf-8 +crawler-price: USD 0.01 +``` + +## 3. Track your spending + +When you successfully access pay per crawl content (response with HTTP status code 200), the response will include `crawler-charged`. For example: + +``` +crawler-charged: USD 0.01 +``` + +Cloudflare strongly recommends tracking and saving these values to keep an accurate record of the bill your AI crawler has accrued. + +## Additional resources + +You may wish to refer to the following resources. + +- [AI Audit FAQs](/ai-audit/reference/faq/). \ No newline at end of file diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/index.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/index.mdx new file mode 100644 index 000000000000000..0e1b55f955524e8 --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/index.mdx @@ -0,0 +1,12 @@ +--- +title: Use pay per crawl as an AI owner +pcx_content_type: navigation +sidebar: + group: + hideIndex: true + order: 5 +--- + +import { DirectoryListing } from "~/components"; + + diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/set-up-cloudflare-account.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/set-up-cloudflare-account.mdx new file mode 100644 index 000000000000000..6fb2323d9d1b29b --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/set-up-cloudflare-account.mdx @@ -0,0 +1,21 @@ +--- +title: Set up your account +pcx_content_type: concept +sidebar: + order: 2 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Set up your
Cloudflare Account]:::highlight --> B[Connect to
Stripe] +B --> C[Verify your
AI crawler] +C --> D[Crawl pages] +classDef highlight fill:#F6821F,color:white +``` + +To begin using pay per crawl, set up your Cloudflare account. + +Sign up for a [Cloudflare account](https://dash.cloudflare.com/sign-up). + diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/verify-ai-crawler.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/verify-ai-crawler.mdx new file mode 100644 index 000000000000000..c96b438f1cf7b5f --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/verify-ai-crawler.mdx @@ -0,0 +1,57 @@ +--- +title: Verify your AI crawler +pcx_content_type: concept +sidebar: + order: 10 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Set up your
Cloudflare Account] --> B[Connect to
Stripe] +B --> C[Verify your
AI crawler]:::highlight +C --> D[Crawl pages] +classDef highlight fill:#F6821F,color:white +``` + +Once you have connected your Stripe account, set up your AI crawler as a [verified bot](/bots/concepts/bot/#verified-bots). + +## Content access restriction + +When an AI crawler tries to access content protected by pay per crawl, it receives a HTTP status code 402. This indicates payment is required. The HTTP header of the response includes the cost of the content. + +For example, the response header may look like below: + +``` +HTTP/2 402 +date: Fri, 06 Jun 2025 08:42:38 GMT +content-type: text/plain; charset=utf-8 +crawler-price: USD 0.01 +server: cloudflare +``` + +To access this content, you must verify your AI crawler. + +## 1. Follow Web Bot Auth protocol + +Ensure your AI crawler identifies itself with the required headers for Web Bot Auth. + +Follow the steps found in [Web Both Auth](/bots/concepts/bot/verified-bots/web-bot-auth/). + +## 2. Follow verified bot policy + +Ensure your AI crawler follows Cloudflare's [verified bots policy](/bots/concepts/bot/verified-bots/policy/). + +## 3. Submit verification request + +Submit a form to add your AI crawler to Cloudflare's list of verified bots. + +{/* prettier-ignore */} + +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account. +2. Go to **Manage Account** > **Configurations**. +3. Go to the **Verified Bots** tab. +4. Fill out the form. +5. Select **Submit**. + diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/connect-to-stripe.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/connect-to-stripe.mdx new file mode 100644 index 000000000000000..2a297c60ecb50a9 --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/connect-to-stripe.mdx @@ -0,0 +1,39 @@ +--- +title: Connect Stripe +pcx_content_type: concept +sidebar: + order: 4 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Set up your
Cloudflare Account] --> B[Connect to
Stripe]:::highlight +B --> C[Set pay per
crawl price] +C --> D[Select AI crawlers
to charge] +D --> E[Monitor
metrics] +classDef highlight fill:#F6821F,color:white +``` + +Connect your Cloudflare account to Stripe to process payments. Pay per crawl uses Stripe to process payments between AI crawler owners and site owners. + +{/* prettier-ignore */} + +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain. +2. Go to **AI Audit**. +3. Go to the **Settings** tab. +4. Under **Pay Per Crawl**, select **Connect**. +5. Select **Continue to Stripe**. +6. Follow the on-screen instructions to connect your Cloudflare account to a Stripe account. + + +When you successfully connect Stripe to your account, you will see a green tick ✅ next to **Stripe connection**. + +## Billing + +Cloudflare manages the billing lifecycle from charge initiation to payout. + +The billing workflow begins when an AI crawler indicates payment intent via request header. A charge event is recorded upon successful delivery of the requested content. Cloudflare subsequently aggregates and reconciles all recorded charges. + +Payouts are remitted to publishers in good standing on a monthly cycle and subject to a settlement period and payout threshold. \ No newline at end of file diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/index.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/index.mdx new file mode 100644 index 000000000000000..96a6dfb68a67d98 --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/index.mdx @@ -0,0 +1,12 @@ +--- +title: Use pay per crawl as a site owner +pcx_content_type: navigation +sidebar: + group: + hideIndex: true + order: 5 +--- + +import { DirectoryListing } from "~/components"; + + diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-metrics.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-metrics.mdx new file mode 100644 index 000000000000000..1ef644f02bf1f35 --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-metrics.mdx @@ -0,0 +1,37 @@ +--- +title: Monitor metrics +pcx_content_type: concept +sidebar: + order: 17 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Set up your
Cloudflare Account] --> B[Connect to
Stripe] +B --> C[Set pay per
crawl price] +C --> D[Select AI crawlers
to charge] +D --> E[Monitor
metrics]:::highlight +classDef highlight fill:#F6821F,color:white +``` + +Once you have confirmed the selection of AI crawlers to charge, monitor key metrics to understand how AI crawlers are interacting with your content. + +{/* prettier-ignore */} + +1. Go to the **Metrics** tab. +2. [View AI Audit metrics](/ai-audit/features/analyze-ai-crawlers/#view-ai-audit-metrics). Understanding these metrics can help you better manage AI crawlers accessing your content. + + +## Other considerations + +You may also wish to do the following: + +1. Update your `robots.txt` to clearly indicate which pages are off-limits, even if AI crawlers are willing to pay for the content. + +## Additional resources + +You may wish to refer to the following resources. + +- [AI Audit FAQs](/ai-audit/reference/faq/). \ No newline at end of file diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge.mdx new file mode 100644 index 000000000000000..5e807cd22175286 --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge.mdx @@ -0,0 +1,29 @@ +--- +title: Select AI crawlers to charge +pcx_content_type: concept +sidebar: + order: 13 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Set up your
Cloudflare Account] --> B[Connect to
Stripe] +B --> C[Set pay per
crawl price] +C --> D[Select AI crawlers
to charge]:::highlight +D --> E[Monitor
metrics] +classDef highlight fill:#F6821F,color:white +``` + +Once you have set a pay per crawl price, AI Audit automatically sets the default action for all applicable AI crawlers as **Charge**. However, you can control which action to take for each AI crawler. + +{/* prettier-ignore */} + +1. Go to the **AI Crawlers** tab. +2. Use the selection slider in the **Actions** column to configure the setting for each AI crawler. + + +AI crawlers for which you select **Charge** will be prompted to pay your configured pay per crawl price each time they attempt to scrape content from a zone. + +Refer to [Manage AI crawlers](/ai-audit/features/manage-ai-crawlers/) for more information on how to manage AI crawlers. diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx new file mode 100644 index 000000000000000..d4609add6fce72b --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx @@ -0,0 +1,32 @@ +--- +title: Set a pay per crawl price +pcx_content_type: concept +sidebar: + order: 10 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Set up your
Cloudflare Account] --> B[Connect to
Stripe] +B --> C[Set pay per
crawl price]:::highlight +C --> D[Select AI crawlers
to charge] +D --> E[Monitor
metrics] +classDef highlight fill:#F6821F,color:white +``` +Once you have connected your Stripe account, you can set your pay per crawl price. + +{/* prettier-ignore */} + +1. From the [Cloudflare dashboard](https://dash.cloudflare.com/), select your account and domain. +2. Go to **AI Audit**. +3. Go to the **Settings** tab. +4. From **Pay Per Crawl**, select the **Edit** icon under **Pay Per Crawl price**. +5. Enter the pay per crawl price you wish to set. Note that smallest unit of payment is 0.01 USD. +6. Select **Save**. + + +:::note +When you configure a pay per crawl price, AI Audit automatically sets the default action for all applicable AI crawlers as **Charge**. +::: \ No newline at end of file diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account.mdx new file mode 100644 index 000000000000000..f696dd381b6ecfd --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account.mdx @@ -0,0 +1,33 @@ +--- +title: Set up your account +pcx_content_type: concept +sidebar: + order: 2 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Set up your
Cloudflare Account]:::highlight --> B[Connect to
Stripe] +B --> C[Set pay per
crawl price] +C --> D[Select AI crawlers
to charge] +D --> E[Monitor
metrics] +classDef highlight fill:#F6821F,color:white +``` + +To begin using pay per crawl, set up your Cloudflare account. + +## 1. Sign up to a Cloudflare account + +Sign up for a [Cloudflare account](https://dash.cloudflare.com/sign-up). + +## 2. Connect your domain to Cloudflare + +[Connect your domain](/fundamentals/manage-domains/add-site/) to Cloudflare. Domains you add to Cloudflare become [zones](/fundamentals/concepts/accounts-and-zones/#zones). + +AI Audit scopes pay per crawl to your zones. + +## 3. Proxy traffic through Cloudflare + +Make sure your domain is [proxying traffic through Cloudflare](/fundamentals/concepts/how-cloudflare-works/#cloudflare-as-a-reverse-proxy). diff --git a/src/content/docs/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl.mdx b/src/content/docs/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl.mdx new file mode 100644 index 000000000000000..f3a8a7e2e8a81a5 --- /dev/null +++ b/src/content/docs/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl.mdx @@ -0,0 +1,44 @@ +--- +title: What is Pay Per Crawl? +pcx_content_type: concept +sidebar: + order: 2 +--- + +import { Steps, GlossaryTooltip } from "~/components"; + +:::note[Closed beta] +Pay per crawl is currently in closed beta. + +To find out how to join the beta program and to learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/). +::: + +AI crawlers often consume vast amounts of web content. Some provide mutual benefit to content owners by indexing content for search engines, but others engage in activities such as content scraping without permission. + +The resulting landscape leaves content owners with limited options for managing AI crawlers or receiving compensation for automated access to their intellectual property. + +## What is Pay Per Crawl? + +Pay per crawl is a feature of AI Audit that enables site owners to control and monetize AI crawler access to content by setting a price per zone. + +Each time an AI crawler requests content, they either present payment intent via request headers for successful `HTTP 200` access, or receive an `HTTP 402 Payment Required` response with pricing. Cloudflare acts as the Merchant of Record. for pay per crawl and also provides the underlying technical infrastructure. + +:::note +If you block an AI crawler from a zone via either of Cloudflare's [WAF](/waf/) or [Bot Management](/bots/get-started/bot-management/) products, those products' rulesets will override pay per crawl's "charge" feature, and the blocked crawler will not have access to the zone. +::: + +Ultimately, pay per crawl enables: + +- Site owners to take control of their content, and charge a fee every time an AI crawler accesses a page in their [Cloudflare zone](/fundamentals/concepts/accounts-and-zones/#zones). For more details, refer to [use pay per crawl as a site owner](/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/). +- AI crawler owners to pay to access content on sites protected by pay per crawl. For more details, refer to [use pay per crawl as an AI owner](/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/). + +![Pay per crawl components](~/assets/images/ai-audit/ai-audit-pay-per-crawl-diagram.png) + +## Additional resources + +Refer to the following resources. + +- [Use pay per crawl as a site owner](/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account/). +- [Use pay per crawl as an AI owner](/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/set-up-cloudflare-account/). +- [AI Audit with Cloudflare WAF](/ai-audit/configuration/ai-audit-with-waf/). +- [AI Audit with Cloudflare Bots](/ai-audit/configuration/ai-audit-with-bots/). \ No newline at end of file diff --git a/src/content/docs/ai-audit/get-started.mdx b/src/content/docs/ai-audit/get-started.mdx index 20298be86e00382..3ec2c2948bf7daf 100644 --- a/src/content/docs/ai-audit/get-started.mdx +++ b/src/content/docs/ai-audit/get-started.mdx @@ -34,9 +34,9 @@ To only block specific AI crawlers: {/* prettier-ignore */} 1. Go to **AI Audit**. -2. From **AI Crawlers** tab, go to the list of AI crawlers. +2. From the **AI Crawlers** tab, go to the list of AI crawlers. 3. Review the AI crawlers that are accessing your domain. -4. From the **Block** column, toggle the switch for AI crawlers you with to block. +4. From the **Block** column, select the option **Block**. :::note[Quality of AI crawler detection] @@ -52,12 +52,12 @@ To only block specific AI crawlers: {/* prettier-ignore */} 1. Go to **AI Audit**. -2. From **AI Crawlers** tab, go to the list of AI crawlers. +2. From the **AI Crawlers** tab, go to the list of AI crawlers. 3. Review key information such as: - Bot operator - Number of requests sent by the AI crawler - Whether the AI crawler is allowed according to your `robots.txt`, or the number of times the AI crawler has violated your `robots.txt`. -4. From the **Block** column, toggle the switch for AI crawlers you with to block. +4. From the **Block** column, toggle the switch for AI crawlers you wish to block. diff --git a/src/content/docs/ai-audit/index.mdx b/src/content/docs/ai-audit/index.mdx index 1cc41e788a95833..797e2384a9b1cf7 100644 --- a/src/content/docs/ai-audit/index.mdx +++ b/src/content/docs/ai-audit/index.mdx @@ -14,13 +14,13 @@ import { Description, Feature, FeatureTable, Plan, LinkButton, RelatedProduct } -Analyze and control third-party AI crawlers in your website. +Analyze and control how third-party AI crawlers access and interact with your website. -AI Audit helps manage AI crawlers on your website by providing visibility on which crawlers are accessing your webpage, and allowing you to block unwanted crawlers. +AI Audit helps manage AI crawlers on your website by providing visibility on which crawlers are accessing your pages, and allowing you to block unwanted crawlers. Get started @@ -35,17 +35,25 @@ AI Audit is currently only available as a beta product. - Allow or block individual AI crawlers. + Control how AI crawlers interact with your domain. - Analyze how AI crawlers interact with your domains. + Gain insight into how AI crawlers are interacting with your pages. + + + + Allow AI crawlers to access content by paying per crawl. --- @@ -66,4 +74,4 @@ Identify and mitigate automated traffic to protect your domain from bad bots. product="waf" > Get automatic protection from vulnerabilities and the flexibility to create custom rules. - \ No newline at end of file + diff --git a/src/content/docs/ai-audit/reference/faq.mdx b/src/content/docs/ai-audit/reference/faq.mdx new file mode 100644 index 000000000000000..2ab80ffbb95924a --- /dev/null +++ b/src/content/docs/ai-audit/reference/faq.mdx @@ -0,0 +1,29 @@ +--- +title: FAQs +pcx_content_type: concept +sidebar: + order: 2 +--- + +import { Steps } from "~/components"; + +## Frequently asked questions - site owners + +### Can I charge a different price for each AI crawlers? + + +No. Pay per crawl allows setting different configurations (Block, Charge, or Allow) for each crawler, but it only supports setting a single price for all crawlers set to "Charge". + +## Frequently asked questions - bot owners + +### Will I be charged multiple times if re-crawling the same page? + +Every time your AI crawler accesses content in a website protected with pay per crawl, it will incur a cost, as set by the site owner. You must implement mechanisms within your crawler to track its expenditure and to enforce any desired spending limits you wish to set. + +### Will I be billed for error HTTP status codes? + +No. Charging events are only issued on successful HTTP response codes. Errors are not billed, even if you have sent the `crawler-exact-price` header or the `crawler-max-price` header. + +### Do I need to use a specific user agent? + +You should use the normal user agents associated with your AI crawler that you have onboarded to Cloudflare and identified using Web Bot Auth. \ No newline at end of file diff --git a/src/content/glossary/ai-audit.yaml b/src/content/glossary/ai-audit.yaml index 526bd02b39a22f9..5c9f7cc25ca3581 100644 --- a/src/content/glossary/ai-audit.yaml +++ b/src/content/glossary/ai-audit.yaml @@ -3,6 +3,12 @@ productName: AI Audit entries: - term: robots.txt general_definition: |- - A text file which lists pages in your website that are off-limits for bots. Well-behaved bots respect this file, but some bots may violate it. You can [enforce robots.txt](/ai-audit/features/enforce-robots-txt/) with Cloudflare WAF custom rules. + A text file which lists pages in your website that are off-limits for bots. Well-behaved bots respect this file, but some bots may violate it. You can enforce `robots.txt` with Cloudflare WAF custom rules. + - term: Merchant of Record + general_definition: |- + The entity who facilitates "buying and selling". For pay per crawl, Cloudflare is the merchant of record. + - term: AI crawler + general_definition: |- + A bot which scrapes content from websites in support of an AI model, including by scraping content for indexing, retrieval augmented generation, or training.