diff --git a/public/__redirects b/public/__redirects index 1c3065c747a2ac..c314a81faff85e 100644 --- a/public/__redirects +++ b/public/__redirects @@ -153,6 +153,12 @@ /ai-audit/features/enforce-robots-txt/ /ai-crawl-control/features/analyze-ai-crawlers/ 301 /ai-audit/features/analyze-ai-crawlers/ /ai-crawl-control/features/analyze-ai-traffic/ 301 +# ai crawl control +/ai-crawl-control/reference/faq/ /ai-crawl-control/features/pay-per-crawl/faq/ 301 +/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account/ /ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings/ 301 +/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/connect-to-stripe/ /ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/ 301 +/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-metrics/ /ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity/ 301 + # AI Gateway /ai-gateway/get-started/configuring-settings/ /ai-gateway/get-started/ 301 /ai-gateway/get-started/connecting-applications/ /ai-gateway/get-started/ 301 @@ -209,7 +215,7 @@ /analytics/analytics-integrations/google-cloud/ /analytics/analytics-integrations/ 301 /analytics/dashboards/ /log-explorer/custom-dashboards/ 301 /analytics/analytics-integrations/looker/ /analytics/analytics-integrations/ 301 -/analytics/network-analytics/reference/network-analytics-v1/ /analytics/graphql-api/migration-guides/network-analytics-v2/ 301 +/analytics/network-analytics/reference/network-analytics-v1/ /analytics/graphql-api/migration-guides/network-analytics-v2/ 301 # email-security /email-security/reporting/search/detection-search/ /email-security/reporting/search/ 301 diff --git a/src/content/docs/ai-crawl-control/features/analyze-ai-traffic.mdx b/src/content/docs/ai-crawl-control/features/analyze-ai-traffic.mdx index e6a87aa85aab2a..f658c510bdda1a 100644 --- a/src/content/docs/ai-crawl-control/features/analyze-ai-traffic.mdx +++ b/src/content/docs/ai-crawl-control/features/analyze-ai-traffic.mdx @@ -16,13 +16,13 @@ To view AI Crawl Control metrics: 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain. 2. Go to **AI Crawl Control**. -You can find meaningful information across both **AI Crawlers** and **Metrics** tabs. +You can find meaningful information across both **Crawlers** and **Metrics** tabs. ## View AI crawler activity -The **AI Crawlers** tab provides you with the following information: +The **Crawlers** tab provides you with the following information: -- Complete list of detected AI crawlers and their operators +- Total number of requests to crawl your website from common AI crawlers - Number of requests made by each AI crawler - Number of `robots.txt` violations for each crawler @@ -30,24 +30,17 @@ The **AI Crawlers** tab provides you with the following information: The **Metrics** tab provides you with the following metrics to help you understand how AI crawlers are interacting with your website. -| Metric | Description | -| ------------------------------------ | ----------------------------------------------------------------------------- | -| Total requests | The total number of requests to crawl your website, from all AI crawlers | -| Blocked requests | The number of crawler requests you have blocked, from any rule | -| Allowed requests | The number of crawler requests you have allowed | -| Hosts | The owner of the AI crawler | -| Overall popular paths | The most popular pages crawled by AI crawlers, from all AI crawlers | -| Most active AI crawlers by operators | The AI crawler owners with the highest number of requests to access your site | -| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler | -| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler | -| Referrals | A graph which displays the number of referrals from each AI operator | -| Referers | The list of referers who directed traffic to your site | - -:::note[Requests in AI Crawl Control metrics] -The number of requests in AI Crawl Control metrics are specifically requests which were met with HTTP code 200 (the request was successfully served, with actual content). - -AI Crawl Control metrics filter all other HTTP codes. -::: +| Metric | Description | +| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Total requests | The total number of requests to crawl your website, from all AI crawlers | +| Allowed requests | The number of crawler requests that received a successful response from your site | +| Unsuccessful requests | The number of crawler requests that failed (HTTP 4xx or 5xx) as a result of a blocked request, other security rules, or website errors such as a crawler attempting to access a non-existent page | +| Overall popular paths | The most popular pages crawled by AI crawlers, from all AI crawlers | +| Most active AI crawlers by operators | The AI crawler owners with the highest number of requests to access your site | +| Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler | +| Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler | +| Referrals | A graph which displays the number of visits that were directed to your site from each AI operator | +| Referers | The list of referers who directed visits to your site | ## Filter date range diff --git a/src/content/docs/ai-crawl-control/features/manage-ai-crawlers.mdx b/src/content/docs/ai-crawl-control/features/manage-ai-crawlers.mdx index 21ca21495adab1..541159a2faaa7a 100644 --- a/src/content/docs/ai-crawl-control/features/manage-ai-crawlers.mdx +++ b/src/content/docs/ai-crawl-control/features/manage-ai-crawlers.mdx @@ -13,22 +13,22 @@ To manage AI crawlers: 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain. 2. Go to **AI Crawl Control**. -3. Go to the **AI Crawlers** tab. +3. Go to the **Crawlers** tab. ## Review AI crawler activity The **Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information. -| Column | Details | -| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Crawler | The name of the AI crawler and the operator that owns it. | -| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/#categories). | -| Requests | Total allowed and blocked requests with trend chart. Blocked requests may come from any configured rule, not just the actions shown here. | -| Robots.txt violations | The number of times the AI crawler has violated your `robots.txt` file. | -| Action | The action you wish to take for the AI crawler. Refer to [Take action for each AI crawler](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). | +| Column | Details | +| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Crawler | The name of the AI crawler and the operator that owns it. | +| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/#categories). | +| Requests | The total number of allowed and unsuccessful requests, with trend chart. Unsuccessful requests may come from any rule or response error, not just the block action in AI Crawl Control. | +| Robots.txt violations | The number of times the AI crawler has violated your `robots.txt` file. | +| Action | The action you wish to take for the AI crawler. Refer to [Take action for each AI crawler](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). | :::note[Quality of AI crawler detection] -On the free plan, AI Crawl Control identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Crawl Control to detect easy-to-detect (well-known) AI crawlers. +On the free plan, AI Crawl Control identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Crawl Control to detect well-known, self-identifying AI crawlers. Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field. ::: @@ -46,31 +46,21 @@ The values of the table will update according to your filter. ## Take action for each AI crawler - - -:::note[Pay per crawl closed beta] -Pay per crawl is currently in closed beta. - -To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer. - -To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/). -::: + -For each AI crawler, you can take one of three actions: allow, charge, or block. +For each AI crawler, you can choose to allow or block access. - **Summary:** You can allow an AI crawler to scrape your content. - **When to use:** Allow AI crawlers that offer services which provide value through citations, referrals, or existing agreements. - **Implementation:** From the **Actions** column, select **Allow**. -Note that you can still choose to [Enforce `robots.txt`](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). -For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works). +Note that you can still choose to [Enforce `robots.txt`](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). - - **Summary:** You can block an AI crawler to completely stop the AI crawler from scraping your webpage. - **When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies. - **Implementation:** From the **Actions** column, select **Block**. @@ -79,32 +69,32 @@ Note that you can configure the response that gets returned when blocking an AI - - -- **Summary:** You can charge the owner of the AI crawler for each crawl request. -- **When to use:** Charge AI crawlers when your content has training value, and you want to explore monetization options -- **Implementation:** From the **Actions** column, select **Charge**. + + -For more information, refer to [What is Pay Per Crawl?](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/). +:::note[Pay per crawl closed beta] +Pay per crawl is currently in closed beta. - +To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer. - - +To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/). +::: -For each AI crawler, you can choose to allow or block access. +For each AI crawler, you can take one of three actions: allow, charge, or block. - **Summary:** You can allow an AI crawler to scrape your content. - **When to use:** Allow AI crawlers that offer services which provide value through citations, referrals, or existing agreements. - **Implementation:** From the **Actions** column, select **Allow**. + Note that you can still choose to [Enforce `robots.txt`](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). -Note that you can still choose to [Enforce `robots.txt`](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). +For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works). + - **Summary:** You can block an AI crawler to completely stop the AI crawler from scraping your webpage. - **When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies. - **Implementation:** From the **Actions** column, select **Block**. @@ -113,9 +103,24 @@ Note that you can configure the response that gets returned when blocking an AI + + +- **Summary:** You can charge the owner of the AI crawler for each successful crawl request. +- **When to use:** Charge AI crawlers when your content has training value, and you want to explore monetization options. +- **Implementation:** From the **Actions** column, select **Charge**. + +For more information, refer to [What is Pay Per Crawl?](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/). + + + + +:::tip[Need more advanced control?] +You can also create more complex rules when taking action on AI crawlers, using [Cloudflare WAF](/waf/). For more information on creating more specific rules, refer to [Create a custom rule in the dashboard](/waf/custom-rules/create-dashboard/). +::: + ## Configure block response @@ -147,8 +152,8 @@ You can choose which HTTP response code to return when blocking an AI crawler. Use the dropdown menu to select the desired response code. You can choose from: - - `403 Forbidden`: Use this option if you wish to indicate that you do not want the AI crawler to access your content. - - `402 Payment Required`: Use this option if you wish to indicate that the AI crawler must pay to access your content. +- `403 Forbidden`: Use this option if you wish to indicate that you do not want the AI crawler to access your content. +- `402 Payment Required`: Use this option if you wish to indicate that the AI crawler must pay to access your content. :::note Behind the scenes, AI Crawl Control uses [Cloudflare WAF](/waf/) to return custom block responses. diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/faq.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/faq.mdx new file mode 100644 index 00000000000000..7c56a01f919f7d --- /dev/null +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/faq.mdx @@ -0,0 +1,28 @@ +--- +title: Pay Per Crawl FAQ +pcx_content_type: concept +sidebar: + order: 4 +--- + +import { Steps } from "~/components"; + +## Frequently asked questions for site owners + +### Can I set different prices for different AI crawlers? + +No. Pay per crawl allows you to configure different actions (Block, Charge, or Allow) for each crawler, but you can only set a single price that applies to all crawlers configured with the "Charge" option. + +## Frequently asked questions for AI bot operators + +### Will I be charged for re-crawling the same page? + +Yes. Every time your AI crawler accesses content on a website protected with pay per crawl, it will incur the cost set by the site owner. You should implement mechanisms within your crawler to track expenditure and enforce any spending limits you want to set. + +### Am I charged for error responses? + +No. Charging events are only triggered for successful HTTP response codes. Error responses are not billed, even if you have sent the `crawler-exact-price` or `crawler-max-price` headers. + +### What user agent should I use? + +Use the standard user agents associated with your AI crawler that you have onboarded to Cloudflare and identified through Web Bot Auth. diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx index 1c02936e537396..d791119d7cb459 100644 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx @@ -85,4 +85,4 @@ Cloudflare strongly recommends tracking and saving these values to keep an accur You may wish to refer to the following resources. -- [AI Crawl Control FAQs](/ai-crawl-control/reference/faq/). \ No newline at end of file +- [Pay Per Crawl FAQs](/ai-crawl-control/features/pay-per-crawl/faq). diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/index.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/index.mdx index 0e1b55f955524e..15503eb3a73ad8 100644 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/index.mdx +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/index.mdx @@ -4,7 +4,7 @@ pcx_content_type: navigation sidebar: group: hideIndex: true - order: 5 + order: 3 --- import { DirectoryListing } from "~/components"; diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/connect-to-stripe.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/connect-to-stripe.mdx deleted file mode 100644 index 749282fe09bdff..00000000000000 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/connect-to-stripe.mdx +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Connect Stripe -pcx_content_type: concept -tags: - - Stripe -sidebar: - order: 4 ---- - -import { Steps } from "~/components"; - -```mermaid -graph LR -A[Set up your
Cloudflare Account] --> B[Connect to
Stripe]:::highlight -B --> C[Set pay per
crawl price] -C --> D[Select AI crawlers
to charge] -D --> E[Monitor
metrics] -classDef highlight fill:#F6821F,color:white -``` - -Connect your Cloudflare account to Stripe to process payments. Pay per crawl uses Stripe to process payments between AI crawler owners and site owners. - -{/* prettier-ignore */} - -1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain. -2. Go to **AI Crawl Control**. -3. Go to the **Settings** tab. -4. Under **Pay Per Crawl**, select **Connect**. -5. Select **Continue to Stripe**. -6. Follow the on-screen instructions to connect your Cloudflare account to a Stripe account. - - -When you successfully connect Stripe to your account, you will see a green tick ✅ next to **Stripe connection**. - -## Billing - -Cloudflare manages the billing lifecycle from charge initiation to payout. - -The billing workflow begins when an AI crawler indicates payment intent via request header. A charge event is recorded upon successful delivery of the requested content. Cloudflare subsequently aggregates and reconciles all recorded charges. - -Payouts are remitted to publishers in good standing on a monthly cycle and subject to a settlement period and payout threshold. \ No newline at end of file diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings.mdx new file mode 100644 index 00000000000000..28749365a59ca5 --- /dev/null +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings.mdx @@ -0,0 +1,48 @@ +--- +title: Enable in account settings +pcx_content_type: how-to +sidebar: + order: 2 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Enable in
account settings]:::highlight --> B[Set a pay per
crawl price ] +B --> C[Select crawlers
to charge] +C --> D[Monitor
activity] +D --> E[Manage
payouts] +classDef highlight fill:#F6821F,color:white + +click B "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price/" +click C "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge/" +click D "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity/" +click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/" +``` + +## Prerequisites + +To configure pay per crawl, you must have the following: + +- **Cloudflare account**: You need an active Cloudflare account with domains added +- **Domain on Cloudflare**: Your domain must be using Cloudflare's nameservers, or have DNS records managed by Cloudflare +- **Administrator access**: You need Administrator or Super Administrator permissions for account-level configuration + +## Configure domain access + +An Administrator or Super Administrator with access to all domains must select which domains should show the pay per crawl controls: + +{/* prettier-ignore */} + +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account. +2. Go to **Manage Account** > **Settings** > **Pay Per Crawl**. +3. In the **Domain Access** table, select which domains will have pay per crawl configurations visible. +4. Set the **Visibility** to **Visible** for each domain you want to configure. + + +:::note[Visibility vs Security] +Setting a domain to **Visible** will not affect security rules. This only makes the pay per crawl configuration controls accessible to domain-level administrators. +::: + +After completing these steps, domain administrators can set a pay per crawl price and enable pay per crawl for their specific domains. diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/index.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/index.mdx index 96a6dfb68a67d9..1cc9b32e1cdd38 100644 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/index.mdx +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/index.mdx @@ -4,7 +4,7 @@ pcx_content_type: navigation sidebar: group: hideIndex: true - order: 5 + order: 2 --- import { DirectoryListing } from "~/components"; diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts.mdx new file mode 100644 index 00000000000000..ddd524282479d4 --- /dev/null +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts.mdx @@ -0,0 +1,59 @@ +--- +title: Manage payouts +pcx_content_type: how-to +tags: + - Stripe +sidebar: + order: 6 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Enable in
account settings] --> B[Set a pay per
crawl price ] +B --> C[Select crawlers
to charge] +C --> D[Monitor
activity] +D --> E[Manage
payouts]:::highlight +classDef highlight fill:#F6821F,color:white + +click A "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings/" +click B "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price/" +click C "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge/" +click D "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity/" +``` + +When you're ready to receive payments for your accrued crawler activity, connect your Cloudflare account to Stripe. This step can be completed at any time after enabling pay per crawl. + +## Create a new Stripe account + +A person with **Administrator** or **Super Administrator** access must set up the Stripe connection: + +{/* prettier-ignore */} + +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account. +2. Go to **Manage Account** > **Settings** > **Pay Per Crawl**. +3. In the **Stripe account** section, select **Connect**. +4. Select **Continue to Stripe**. +5. Complete Stripe's onboarding process, including: + - Basic business information + - Bank account details for payouts + + +:::note[Pay Per Crawl Stripe account required] +You must create a dedicated Cloudflare Stripe Connect account through the dashboard. Pre-existing Stripe accounts are not compatible with this feature. +::: + +## Billing lifecycle + +Cloudflare manages the complete billing lifecycle: + +1. **Charge initiation**: AI crawlers indicate payment intent via request headers +2. **Charge recording**: A charge event is recorded upon successful content delivery (HTTP 200 response) +3. **Aggregation**: Cloudflare aggregates and reconciles all recorded charges +4. **Payout**: Monthly payments to publishers in good standing + +### Limitations + +- Your accrued balance is not currently visible in the dashboard. You can request balance updates from your Cloudflare team. +- Payouts are subject to settlement periods and minimum payout thresholds. diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity.mdx new file mode 100644 index 00000000000000..499226efb9a5b2 --- /dev/null +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity.mdx @@ -0,0 +1,66 @@ +--- +title: Monitor activity +pcx_content_type: how-to +sidebar: + order: 5 +--- + +import { Steps } from "~/components"; + +```mermaid +graph LR +A[Enable in
account settings] --> B[Set a pay per
crawl price ] +B --> C[Select crawlers
to charge] +C --> D[Monitor
activity]:::highlight +D --> E[Manage
payouts] +classDef highlight fill:#F6821F,color:white + +click A "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings/" +click B "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price/" +click C "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge/" +click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/" +``` + +After configuring pay per crawl, monitor crawler activity to understand how AI crawlers interact with your content, and track your earnings. + +## View crawler activity + +{/* prettier-ignore */} + +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain. +2. Go to **AI Crawl Control**. +3. Go to the **Metrics** tab to view detailed analytics. + + +The metrics help you understand: + +- Which crawlers are accessing your content +- How often they are being charged +- Request patterns and trends +- Robots.txt violations + +For detailed information about available metrics, refer to [View AI Crawl Control metrics](/ai-crawl-control/features/analyze-ai-traffic/#view-ai-crawl-control-metrics). + +:::note[Balance visibility] +Your accrued earnings balance is not currently visible in the dashboard. You can request balance updates from your Cloudflare team. +::: + +## Additional considerations + +### Robots.txt management + +Consider updating your `robots.txt` file to clearly indicate which pages should remain off-limits, even if AI crawlers are willing to pay for access. + +### Ongoing optimization + +Do the following to ensure you are using pay per crawl most effectively: + +- Review crawler activity regularly to identify patterns +- Adjust pricing based on demand and content value +- Modify crawler actions (charge, allow, block) as needed +- Monitor for any unusual or unwanted crawler behavior + +## Additional resources + +- [Pay Per Crawl FAQs](/ai-crawl-control/features/pay-per-crawl/faq) +- [Analyze AI traffic](/ai-crawl-control/features/analyze-ai-traffic/) diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-metrics.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-metrics.mdx deleted file mode 100644 index 9c11fcb1b07e54..00000000000000 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-metrics.mdx +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: Monitor metrics -pcx_content_type: concept -sidebar: - order: 17 ---- - -import { Steps } from "~/components"; - -```mermaid -graph LR -A[Set up your
Cloudflare Account] --> B[Connect to
Stripe] -B --> C[Set pay per
crawl price] -C --> D[Select AI crawlers
to charge] -D --> E[Monitor
metrics]:::highlight -classDef highlight fill:#F6821F,color:white -``` - -Once you have confirmed the selection of AI crawlers to charge, monitor key metrics to understand how AI crawlers are interacting with your content. - -{/* prettier-ignore */} - -1. Go to the **Metrics** tab. -2. [View AI Crawl Control metrics](/ai-crawl-control/features/analyze-ai-traffic/#view-ai-audit-metrics). Understanding these metrics can help you better manage AI crawlers accessing your content. - - -## Other considerations - -You may also wish to do the following: - -1. Update your `robots.txt` to clearly indicate which pages are off-limits, even if AI crawlers are willing to pay for the content. - -## Additional resources - -You may wish to refer to the following resources. - -- [AI Crawl Control FAQs](/ai-crawl-control/reference/faq/). \ No newline at end of file diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge.mdx index d83e248b6d2501..f126acfeba12c0 100644 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge.mdx +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge.mdx @@ -1,29 +1,53 @@ --- -title: Select AI crawlers to charge -pcx_content_type: concept +title: Select crawlers to charge +pcx_content_type: how-to sidebar: - order: 13 + order: 4 --- import { Steps } from "~/components"; ```mermaid graph LR -A[Set up your
Cloudflare Account] --> B[Connect to
Stripe] -B --> C[Set pay per
crawl price] -C --> D[Select AI crawlers
to charge]:::highlight -D --> E[Monitor
metrics] +A[Enable in
account settings] --> B[Set a pay per
crawl price ] +B --> C[Select crawlers
to charge]:::highlight +C --> D[Monitor
activity] +D --> E[Manage
payouts] classDef highlight fill:#F6821F,color:white + +click A "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings/" +click B "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price/" +click D "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity/" +click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/" ``` -Once you have set a pay per crawl price, AI Crawl Control automatically sets the default action for all applicable AI crawlers as **Charge**. However, you can control which action to take for each AI crawler. +Once you have enabled pay per crawl and set a price, you can specify which AI crawlers to charge for accessing your content. {/* prettier-ignore */} -1. Go to the **AI Crawlers** tab. -2. Use the selection slider in the **Actions** column to configure the setting for each AI crawler. +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain. +2. Go to **AI Crawl Control**. +3. Go to the **Crawlers** tab. +4. For each crawler, choose an action from the **Actions** column: + - **Charge**: Charge the set price for successful content access + - **Allow**: Allow free access without charging + - **Block**: Block access completely -AI crawlers for which you select **Charge** will be prompted to pay your configured pay per crawl price each time they attempt to scrape content from a zone. +:::tip[Search Engine Crawlers and SEO] +Use the **Category** column to identify which bots are **Search Engine Crawlers**. Setting these crawlers to **Block** or **Charge** may negatively impact your site's SEO performance, as search engines may not be able to properly index your content. +::: + +## Bulk actions + +To configure multiple crawlers at once: + +{/* prettier-ignore */} + +1. Use the filters (Name, Operator, Category) to narrow down the crawler list. +2. Select the crawlers you want to configure by checking their boxes. +3. Bulk action options will appear above the table. +4. Select the desired action and apply the changes. + -Refer to [Manage AI crawlers](/ai-crawl-control/features/manage-ai-crawlers/) for more information on how to manage AI crawlers. +For more information on managing AI crawlers, refer to [Manage AI crawlers](/ai-crawl-control/features/manage-ai-crawlers/). diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx index b0fc6bf86b82ac..45d997e9faa645 100644 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx @@ -1,32 +1,40 @@ --- title: Set a pay per crawl price -pcx_content_type: concept +pcx_content_type: how-to sidebar: - order: 10 + order: 3 --- import { Steps } from "~/components"; ```mermaid graph LR -A[Set up your
Cloudflare Account] --> B[Connect to
Stripe] -B --> C[Set pay per
crawl price]:::highlight -C --> D[Select AI crawlers
to charge] -D --> E[Monitor
metrics] +A[Enable in
account settings] --> B[Set a pay per
crawl price ]:::highlight +B --> C[Select crawlers
to charge] +C --> D[Monitor
activity] +D --> E[Manage
payouts] classDef highlight fill:#F6821F,color:white + +click A "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings/" +click C "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge/" +click D "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity/" +click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/" ``` -Once you have connected your Stripe account, you can set your pay per crawl price. + +Once your domain's visibility is set to **Visible** in Account Settings, you can set a pay per crawl price and enable pay per crawl for that domain. {/* prettier-ignore */} -1. From the [Cloudflare dashboard](https://dash.cloudflare.com/), select your account and domain. +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain. 2. Go to **AI Crawl Control**. 3. Go to the **Settings** tab. -4. From **Pay Per Crawl**, select the **Edit** icon under **Pay Per Crawl price**. -5. Enter the pay per crawl price you wish to set. Note that smallest unit of payment is 0.01 USD. +4. In the **Pay Per Crawl** card, select **Enable**. +5. Set your per crawl price - this is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler. 6. Select **Save**. -:::note -When you configure a pay per crawl price, AI Crawl Control automatically sets the default action for all applicable AI crawlers as **Charge**. -::: \ No newline at end of file +After enabling and setting a price, the domain's status in Account Settings will change to **Enabled**. + +:::note[Pricing considerations] +The minimum price is $0.01 USD per crawl. Consider your content value and expected crawler volume when setting your price. +::: diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account.mdx deleted file mode 100644 index 35d9a2e39b83c0..00000000000000 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account.mdx +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Set up your account -pcx_content_type: concept -sidebar: - order: 2 ---- - -import { Steps } from "~/components"; - -```mermaid -graph LR -A[Set up your
Cloudflare Account]:::highlight --> B[Connect to
Stripe] -B --> C[Set pay per
crawl price] -C --> D[Select AI crawlers
to charge] -D --> E[Monitor
metrics] -classDef highlight fill:#F6821F,color:white -``` - -To begin using pay per crawl, set up your Cloudflare account. - -## 1. Sign up to a Cloudflare account - -Sign up for a [Cloudflare account](https://dash.cloudflare.com/sign-up). - -## 2. Connect your domain to Cloudflare - -[Connect your domain](/fundamentals/manage-domains/add-site/) to Cloudflare. Domains you add to Cloudflare become [zones](/fundamentals/concepts/accounts-and-zones/#zones). - -AI Crawl Control scopes pay per crawl to your zones. - -## 3. Proxy traffic through Cloudflare - -Make sure your domain is [proxying traffic through Cloudflare](/fundamentals/concepts/how-cloudflare-works/#cloudflare-as-a-reverse-proxy). - -:::note[Pay per crawl closed beta] -Pay per crawl is currently in closed beta. - -To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer. - -To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/). -::: \ No newline at end of file diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl.mdx index 82b4ea73ed0141..baaca6288f7557 100644 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl.mdx +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl.mdx @@ -2,7 +2,7 @@ title: What is Pay Per Crawl? pcx_content_type: concept sidebar: - order: 2 + order: 1 --- import { Steps, GlossaryTooltip } from "~/components"; @@ -40,7 +40,7 @@ Ultimately, pay per crawl enables: Refer to the following resources. -- [Use pay per crawl as a site owner](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account/). +- [Use pay per crawl as a site owner](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings/). - [Use pay per crawl as an AI owner](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/set-up-cloudflare-account/). - [AI Crawl Control with Cloudflare WAF](/ai-crawl-control/configuration/ai-crawl-control-with-waf/). - [AI Crawl Control with Cloudflare Bots](/ai-crawl-control/configuration/ai-crawl-control-with-bots/). diff --git a/src/content/docs/ai-crawl-control/get-started.mdx b/src/content/docs/ai-crawl-control/get-started.mdx index 9f0a016263b4f7..b8142c9f5d6977 100644 --- a/src/content/docs/ai-crawl-control/get-started.mdx +++ b/src/content/docs/ai-crawl-control/get-started.mdx @@ -34,13 +34,13 @@ To only block specific AI crawlers: {/* prettier-ignore */} 1. Go to **AI Crawl Control**. -2. From the **AI Crawlers** tab, go to the list of AI crawlers. +2. From the **Crawlers** tab, go to the list of AI crawlers. 3. Review the AI crawlers that are accessing your domain. 4. From the **Block** column, select the option **Block**. :::note[Quality of AI crawler detection] -On the free plan, AI Crawl Control identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Crawl Control to detect easy-to-detect (well-known) AI crawlers. +On the free plan, AI Crawl Control identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Crawl Control to detect well-known, self-identifying AI crawlers. Upgrade your plan to enable a more thorough detection using Cloudflare's [Bot Management detection ID](/bots/reference/bot-management-variables/#ruleset-engine-fields) field. ::: @@ -52,7 +52,7 @@ To only block specific AI crawlers: {/* prettier-ignore */} 1. Go to **AI Crawl Control**. -2. From the **AI Crawlers** tab, go to the list of AI crawlers. +2. From the **Crawlers** tab, go to the list of AI crawlers. 3. Review key information such as: - Bot operator - Number of requests sent by the AI crawler diff --git a/src/content/docs/ai-crawl-control/reference/faq.mdx b/src/content/docs/ai-crawl-control/reference/faq.mdx deleted file mode 100644 index 2ab80ffbb95924..00000000000000 --- a/src/content/docs/ai-crawl-control/reference/faq.mdx +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: FAQs -pcx_content_type: concept -sidebar: - order: 2 ---- - -import { Steps } from "~/components"; - -## Frequently asked questions - site owners - -### Can I charge a different price for each AI crawlers? - - -No. Pay per crawl allows setting different configurations (Block, Charge, or Allow) for each crawler, but it only supports setting a single price for all crawlers set to "Charge". - -## Frequently asked questions - bot owners - -### Will I be charged multiple times if re-crawling the same page? - -Every time your AI crawler accesses content in a website protected with pay per crawl, it will incur a cost, as set by the site owner. You must implement mechanisms within your crawler to track its expenditure and to enforce any desired spending limits you wish to set. - -### Will I be billed for error HTTP status codes? - -No. Charging events are only issued on successful HTTP response codes. Errors are not billed, even if you have sent the `crawler-exact-price` header or the `crawler-max-price` header. - -### Do I need to use a specific user agent? - -You should use the normal user agents associated with your AI crawler that you have onboarded to Cloudflare and identified using Web Bot Auth. \ No newline at end of file